vo_opengl: let hwdec driver report the exact image format

Hardware decoding/displaying with vo_opengl is done by replacing the
normal video textures with textures provided by the hardware decoding
API OpenGL interop code. Often, this changes the format (vaglx and vdpau
return RGBA, vda returns packed YUV).

If the format is changed, there was a chance (or at least a higher
potential for bugs) that the shader generation code could be confused by
the mismatch of formats, and would create incorrect conversions.

Simplify this by requiring the hwdec interop driver to set the format it
will return to us. This affects all fields, not just some (done by
replacing the format with the value of the converted_imgfmt field in
init_format), in particular fields like colorlevels.

Currently, no hwdec interop driver does anything sophisticated, and the
win is mostly from the mp_image_params_guess_csp() function, which will
reset fields like colorlevels to expected value if RGBA is used.

1 files changed, 2 insertions, 1 deletions

diff --git a/video/out/gl_hwdec.h b/video/out/gl_hwdec.h
index d4e500e329..cd2ab26892 100644
--- a/video/out/gl_hwdec.h
+++ b/video/out/gl_hwdec.h
@@ -35,7 +35,8 @@ struct gl_hwdec_driver {
     int (*create)(struct gl_hwdec *hw);
     // Prepare for rendering video. (E.g. create textures.)
     // Called on initialization, and every time the video size changes.
-    int (*reinit)(struct gl_hwdec *hw, const struct mp_image_params *params);
+    // *params must be set to the format the hw textures return.
+    int (*reinit)(struct gl_hwdec *hw, struct mp_image_params *params);
     // Return textures that contain the given hw_image.
     // Note that the caller keeps a reference to hw_image until unmap_image
     // is called, so the hwdec driver doesn't need to do that.