vo_opengl: let hwdec driver report the exact image format

Hardware decoding/displaying with vo_opengl is done by replacing the
normal video textures with textures provided by the hardware decoding
API OpenGL interop code. Often, this changes the format (vaglx and vdpau
return RGBA, vda returns packed YUV).

If the format is changed, there was a chance (or at least a higher
potential for bugs) that the shader generation code could be confused by
the mismatch of formats, and would create incorrect conversions.

Simplify this by requiring the hwdec interop driver to set the format it
will return to us. This affects all fields, not just some (done by
replacing the format with the value of the converted_imgfmt field in
init_format), in particular fields like colorlevels.

Currently, no hwdec interop driver does anything sophisticated, and the
win is mostly from the mp_image_params_guess_csp() function, which will
reset fields like colorlevels to expected value if RGBA is used.

1 files changed, 3 insertions, 1 deletions

diff --git a/video/out/gl_hwdec_vaglx.c b/video/out/gl_hwdec_vaglx.c
index f9dc42911d..99d29dc385 100644
--- a/video/out/gl_hwdec_vaglx.c
+++ b/video/out/gl_hwdec_vaglx.c
@@ -87,7 +87,7 @@ static int create(struct gl_hwdec *hw)
     return 0;
 }
 
-static int reinit(struct gl_hwdec *hw, const struct mp_image_params *params)
+static int reinit(struct gl_hwdec *hw, struct mp_image_params *params)
 {
     struct priv *p = hw->priv;
     GL *gl = hw->gl;
@@ -95,6 +95,8 @@ static int reinit(struct gl_hwdec *hw, const struct mp_image_params *params)
 
     destroy_texture(hw);
 
+    params->imgfmt = hw->driver->imgfmt;
+
     gl->GenTextures(1, &p->gl_texture);
     gl->BindTexture(GL_TEXTURE_2D, p->gl_texture);
     gl->TexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);