hwtransfer: check if the source format is accepted directly by the VO

This may seem obvious in retrospect, but we need to explicitly account
for the case where the source format is supported by the VO, but not
a valid target format for the conversion filter. In that situation, we
would conclude that conversion was necessary, because we relied solely
on the conversion filter to identify acceptable target formats.

To avoid that, we should go through the VO's reported set of supported
formats and if the source format is on the list, ensure that format is
also on the target list.

This is mostly a no-op as most VOs do not report supported formats
(instead assuming that all formats decoders can produce are supported)
and in the case where it matters (vaapi), there is only one format that
the VO supports which the conversion filter does not (yuv444p).

1 files changed, 26 insertions, 0 deletions

diff --git a/filters/f_hwtransfer.c b/filters/f_hwtransfer.c
index 9e1e06e188..f9d0458e50 100644
--- a/filters/f_hwtransfer.c
+++ b/filters/f_hwtransfer.c
@@ -367,6 +367,32 @@ static bool probe_formats(struct mp_filter *f, int hw_imgfmt, bool use_conversio
 
             p->fmt_upload_index[index] = p->num_upload_fmts;
 
+            /*
+             * First check if the VO supports the source format. If it does,
+             * ensure it is in the target list, so that we never do an
+             * unnecessary conversion. This explicit step is required because
+             * there can be situations where the conversion filter cannot output
+             * the source format, but the VO can accept it, so just looking at
+             * the supported conversion targets can make it seem as if a
+             * conversion is required.
+             */
+            if (!ctx->supported_formats) {
+                /*
+                 * If supported_formats is unset, that means we should assume
+                 * the VO can accept all source formats, so append the source
+                 * format.
+                 */
+                MP_TARRAY_APPEND(p, p->upload_fmts, p->num_upload_fmts, imgfmt);
+            } else {
+                for (int i = 0; ctx->supported_formats[i]; i++) {
+                    int fmt = ctx->supported_formats[i];
+                    if (fmt == imgfmt) {
+                        MP_VERBOSE(f, "  vo accepts %s\n", mp_imgfmt_to_name(fmt));
+                        MP_TARRAY_APPEND(p, p->upload_fmts, p->num_upload_fmts, fmt);
+                    }
+                }
+            }
+
             enum AVPixelFormat *fmts = conversion_cstr->valid_sw_formats;
             for (int i = 0; fmts && fmts[i] != AV_PIX_FMT_NONE; i++) {
                 int fmt = pixfmt2imgfmt(fmts[i]);