hwdec/dmabuf_interop_gl: work-around implicit rgba swizzling

This has been a long standing problem that I think was the root cause
for wm4's vaapi shitlist. As DRM explicit supports all possible rgba
orderings, but OpenGL only guarantees support for 'rgba8', something
has to happen to support import of dmabufs with non-native ordering.

Although undocumented, it appears that somewhere in the import on the
GL side, the ordering is swizzled to rgba - which makes sense as that's
the only way to be able to support importing all the different formats.

But we didn't know this, and we do our own swizzling based on the
original imgfmt - which matches the drm format. So we get double
swizzling and messed up colours.

As there is no way for us to express that the GL swizzle happened to
the rest of mpv, the easiest thing to do is fudge the format to the
natural ordering so the GL doesn't change anything, then our swizzling
will do the right thing in the end.

Note that this change doesn't handle 0rgb and 0bgr because they seem to
be affected by some additional feature/bug that is mangling the
ordering somewhere that equally affects Vulkan interop. That could be a
vaapi driver bug or something going on in ffmpeg.

1 files changed, 33 insertions, 0 deletions

diff --git a/video/out/hwdec/dmabuf_interop_gl.c b/video/out/hwdec/dmabuf_interop_gl.c
index ee9245fb72..2d83c1c096 100644
--- a/video/out/hwdec/dmabuf_interop_gl.c
+++ b/video/out/hwdec/dmabuf_interop_gl.c
@@ -190,6 +190,39 @@ static bool vaapi_gl_map(struct ra_hwdec_mapper *mapper,
                        p_mapper->desc.layers[i].format);
                 return false;
             }
+        } else {
+            /*
+             * As OpenGL only has one guaranteed rgba format (rgba8), drivers
+             * that support importing dmabuf formats with different channel
+             * orders do implicit swizzling to get to rgba. However, we look at
+             * the original imgfmt to decide channel order, and we then swizzle
+             * based on that. So, we can get into a situation where we swizzle
+             * twice and end up with a mess.
+             *
+             * The simplest way to avoid that is to lie to OpenGL and say that
+             * the surface we are importing is in the natural channel order, so
+             * that our swizzling does the right thing.
+             *
+             * DRM ABGR corresponds to OpenGL RGBA due to different naming
+             * conventions.
+             */
+            switch (format[0]) {
+            case DRM_FORMAT_ARGB8888:
+            case DRM_FORMAT_RGBA8888:
+            case DRM_FORMAT_BGRA8888:
+                format[0] = DRM_FORMAT_ABGR8888;
+                break;
+            case DRM_FORMAT_XRGB8888:
+                format[0] = DRM_FORMAT_XBGR8888;
+                break;
+            case DRM_FORMAT_RGBX8888:
+            case DRM_FORMAT_BGRX8888:
+                // Logically, these two formats should be handled as above,
+                // but there appear to be additional problems that make the
+                // format change here insufficient or incorrect, so we're
+                // doing nothing for now.
+                break;
+            }
         }
 
         for (int j = 0; j < p_mapper->desc.layers[i].nb_planes; j++, n++) {