f_hwtransfer: disable vulkan multiplane images when uploading from cuda

Although we can support vulkan multiplane images, cuda lacks any such
support, and so cannot natively import such images for interop. It's
possible that we can do separate exports for each plane in the image
and have it work, but for now, we can selectively disable multiplane
when we know that we'll be consuming cuda frames.

As a reminder, even though cuda is the frame source, interop is one way
so the vulkan images have to be imported to cuda before we copy the
frame contents over.

This logic here is slightly more complex than I'd like but you can't
just set the flag blindly, as it will cause hwframes ctx creation to
fail if the format is packed or if it's planar rgb. Oh well.

1 files changed, 3 insertions, 2 deletions

diff --git a/filters/f_hwtransfer.c b/filters/f_hwtransfer.c
index 9488c5536c..b281a15838 100644
--- a/filters/f_hwtransfer.c
+++ b/filters/f_hwtransfer.c
@@ -198,7 +198,8 @@ static void process(struct mp_filter *f)
     }
 
     if (!mp_update_av_hw_frames_pool(&p->hw_pool, p->av_device_ctx, p->hw_imgfmt,
-                                     p->last_hw_output_fmt, src->w, src->h))
+                                     p->last_hw_output_fmt, src->w, src->h,
+                                     src->imgfmt == IMGFMT_CUDA))
     {
         MP_ERR(f, "failed to create frame pool\n");
         goto error;
@@ -353,7 +354,7 @@ static bool probe_formats(struct mp_hwupload *u, int hw_imgfmt)
         // Creates an AVHWFramesContexts with the given parameters.
         AVBufferRef *frames = NULL;
         if (!mp_update_av_hw_frames_pool(&frames, ctx->av_device_ref,
-                                         hw_imgfmt, imgfmt, 128, 128))
+                                         hw_imgfmt, imgfmt, 128, 128, false))
         {
             MP_WARN(u->f, "failed to allocate pool\n");
             continue;