hwtransfer: use the right hardware config to find conversion targets

The last piece in the puzzle for doing hardware conversions
automatically is ensuring we only consider valid target formats for the
conversion. Although it is unintuitive, some vaapi drivers can expose a
different set of formats for uploads vs for conversions, and that is
the case on the Intel hardware I have here.

Before this change, we would use the upload target list, and our
selection algorithm would pick a format that doesn't work for
conversions, causing everything to fail. Whoops.

Successfully obtaining the conversion target format list is a bit of a
convoluted process, with only parts of it encapsulated by ffmpeg.
Specifically, ffmpeg understands the concept of hardware configurations
that can affect the constraints of a device, but does not define what
configurations are - that is left up to the specific hwdevice type.

In the case of vaapi, we need to create a config for the video
processing endpoint, and use that when querying for constraints.

I decided to encapsulate creation of the config as part of the hwdec
init process, so that the constraint query can be down in the
hwtransfer code in an opaque way. I don't know if any other hardware
will need this capability, but if so, we'll be able to account for it.

Then, when we look at probing, instead of checking for what formats
are supported for transfers, we use the results of the constraint query
with the conversion config. And as that config doesn't depend on the
source format, we only need to do it once.

1 files changed, 4 insertions, 0 deletions

diff --git a/video/hwdec.h b/video/hwdec.h
index 44008dece0..723c60f427 100644
--- a/video/hwdec.h
+++ b/video/hwdec.h
@@ -23,6 +23,10 @@ struct mp_hwdec_ctx {
     // This will be used for hardware conversion of frame formats.
     // NULL otherwise.
     const char *conversion_filter_name;
+
+    // The libavutil hwconfig to be used when querying constraints for the
+    // conversion filter. Can be NULL if no special config is required.
+    void *conversion_config;
 };
 
 // Used to communicate hardware decoder device handles from VO to video decoder.