videotoolbox: change how videotoolbox format is managed

The underlying intention of this code is to make changing
--videotoolbox-format at runtime work. For this reason, the format can't
just be statically setup, but must be read from the option at runtime.

This means the format is not fixed anymore, and we have to make sure the
renderer is property reinitialized if the format changes. There is
currently no way to trigger reinit on this level, which is why the
mp_image_params.hw_subfmt field was introduced.

One sketchy thing remains: normally, the renderer is supposed to be
involved with VO format negotiation, which would ensure that the VO
can take the format at all. Since the hw_subfmt is not part of this
format negotiation, it's implied the get_vt_fmt() callback only
returns formats supported by the renderer. This is not necessarily
clear because vo_opengl checks this with converted_imgfmt separately.
None of this matters in practice though, because we know all formats
are always supported.

(This still requires somehow triggering decoder reinit to make the
change effective.)

1 files changed, 1 insertions, 0 deletions

diff --git a/video/hwdec.h b/video/hwdec.h
index 9585e32e25..898b035361 100644
--- a/video/hwdec.h
+++ b/video/hwdec.h
@@ -32,6 +32,7 @@ struct mp_hwdec_ctx {
     struct mp_vdpau_ctx *vdpau_ctx;
     struct mp_vaapi_ctx *vaapi_ctx;
     struct mp_d3d_ctx *d3d_ctx;
+    uint32_t (*get_vt_fmt)(struct mp_hwdec_ctx *ctx);
 
     // Optional.
     // Allocates a software image from the pool, downloads the hw image from