video: fix VideoToolbox/VDA autodetection

This affects vo_opengl_cb in particular: it'll most likely auto-load
VDA, and then the VideoToolbox decoder won't work. And everything fails.

This is mainly caused by FFmpeg using separate pixfmts for the _same_
thing (CVPixelBuffers), simply because libavcodec's architecture demands
that hwaccel backends are selected by pixfmts. (Which makes no sense,
but now we have the mess.)

So instead of duplicating FFmpeg's misdesign, just change the format to
our own canonical one on the image output by the decoder. Now the GL
interop code is exactly the same for VDA and VT, and we use the VT name
only.

1 files changed, 5 insertions, 2 deletions

diff --git a/video/img_format.h b/video/img_format.h
index 77d722f7c9..f88592e7f9 100644
--- a/video/img_format.h
+++ b/video/img_format.h
@@ -202,11 +202,14 @@ enum mp_imgfmt {
     // structures, instead of pixel data.
     IMGFMT_VDPAU,           // VdpVideoSurface
     IMGFMT_VDPAU_OUTPUT,    // VdpOutputSurface
-    IMGFMT_VDA,
-    IMGFMT_VIDEOTOOLBOX,
     IMGFMT_VAAPI,
     IMGFMT_DXVA2,           // IDirect3DSurface9 (NV12)
     IMGFMT_MMAL,            // MMAL_BUFFER_HEADER_T
+    // These use the same underlying format, but FFmpeg requires us to keep
+    // them separate. The VDA decoder will change the format to
+    // IMGFMT_VIDEOTOOLBOX, though.
+    IMGFMT_VIDEOTOOLBOX,
+    IMGFMT_VDA,
 
     // Generic pass-through of AV_PIX_FMT_*. Used for formats which don't have
     // a corresponding IMGFMT_ value.