vo_opengl, vaapi: properly probe 10 bit rendering support

There are going to be users who have a Mesa installation which do not
support 10 bit, but a GPU which can decode to 10 bit. So it's probably
better not to hardcode whether it is supported.

Introduce a more general way to signal supported formats from renderer
to decoder. Obviously this is imperfect, because it still isn't part of
proper format negotation (for example, what if there's a vavpp filter,
which accepts anything). Still slightly better than before.

I don't know any way to probe for vaapi dmabuf/EGL dmabuf support
properly (in particular testing specific formats, not just general
availability). So we stay with the current approach and try to create
and map dummy surfaces on init to probe for support. Overdo it and check
all formats that AVHWFramesConstraints reports, instead of only NV12 and
P010 surfaces.

Since we can support unknown formats now, add explicitly checks to the
EGL/dmabuf mapper code to reject unsupported formats. I also noticed
that libavutil signals support for RGB0/BGR0, but couldn't get it to
work. Remove the DRM formats that are unused/didn't work the way I tried
to use them.

With this, 10 bit decoding + rendering should work, provided you have
a capable CPU and a patched Mesa. The required Mesa patch adds support
for the R16 and GR32 formats. It was sent by a Kodi developer to the
Mesa developer mailing list and was not accepted yet.

1 files changed, 5 insertions, 1 deletions

diff --git a/video/hwdec.h b/video/hwdec.h
index f2fa7943af..2d088399f1 100644
--- a/video/hwdec.h
+++ b/video/hwdec.h
@@ -47,7 +47,11 @@ struct mp_hwdec_ctx {
     //  HWDEC_CUDA:             CUcontext*
     void *ctx;
 
-    // Optional.
+    // List of IMGFMT_s, terminated with 0. NULL if N/A.
+    int *supported_formats;
+
+    // Optional. Legacy. (New code should use AVHWFramesContext and
+    // mp_image_hw_download().)
     // Allocates a software image from the pool, downloads the hw image from
     // mpi, and returns it.
     // pool can be NULL (then just use straight allocation).