vo_opengl: vdpau interop without RGB conversion

Until now, we've always converted vdpau video surfaces to RGB, and then
mapped the resulting RGB texture. Change this so that the surface is
mapped as NV12 plane textures.

The reason this wasn't done until now is because vdpau surfaces are
mapped in an "interlaced" way as separate fields, even for progressive
video. This requires messy reinterleraving. It turns out that even
though it's an extra processing step, the result can be faster than
going through the video mixer for RGB conversion.

Other than some potential speed-gain, doing this has multiple other
advantages. We can apply our own color conversion, which is important in
more complex cases. We can correctly apply debanding and potentially
other processing that requires chroma-specific or in-YUV handling.

If deinterlacing is enabled, this switches back to the old RGB
conversion method. Until we have at least a primitive deinterlacer in
vo_opengl, this will stay this way. The d3d11 and vaapi code paths are
similar. (Of course these don't require any crazy field reinterleaving.)

1 files changed, 12 insertions, 0 deletions

diff --git a/video/decode/vdpau.c b/video/decode/vdpau.c
index 2aba10c13b..0003182dcb 100644
--- a/video/decode/vdpau.c
+++ b/video/decode/vdpau.c
@@ -61,6 +61,17 @@ static struct mp_image *allocate_image(struct lavc_ctx *ctx, int w, int h)
     return mp_vdpau_get_video_surface(p->mpvdp, chroma, s_w, s_h);
 }
 
+static struct mp_image *update_format(struct lavc_ctx *ctx, struct mp_image *img)
+{
+    VdpChromaType chroma = 0;
+    uint32_t s_w, s_h;
+    if (av_vdpau_get_surface_parameters(ctx->avctx, &chroma, &s_w, &s_h) >= 0) {
+        if (chroma == VDP_CHROMA_TYPE_420)
+            img->params.hw_subfmt = IMGFMT_NV12;
+    }
+    return img;
+}
+
 static void uninit(struct lavc_ctx *ctx)
 {
     struct priv *p = ctx->hwdec_priv;
@@ -99,4 +110,5 @@ const struct vd_lavc_hwdec mp_vd_lavc_vdpau = {
     .uninit = uninit,
     .init_decoder = init_decoder,
     .allocate_image = allocate_image,
+    .process_image = update_format,
 };