summaryrefslogtreecommitdiffstats
path: root/filters/f_hwtransfer.c
Commit message (Collapse)AuthorAgeFilesLines
* video: don't define IMGFMT_VULKAN conditionallysfan52024-02-261-2/+0
| | | | | | We generally try to avoid that due to the #ifdef mess. The equivalent format is defined in ffmpeg 4.4 while our interop code requires a higher version, but that doesn't cause any problems.
* hwtransfer: actually treat hardware formats as supported input formatsPhilip Langdale2023-12-151-3/+5
| | | | | | | | | | | | | It's conceptually broken for an hw format to be identified as a supported sw format for an hwdec, but in the case of the drm hwdec, we have no choice but to do so for the weird rpi formats, as these are declared, incorrectly, as hw formats in their forked ffmpegs. This means we can't reject such formats as a matter of principle, as we do today. Instead let's just assume that such formats can always be accepted as-is, and will never require conversion. In practice, this is either true or it will fail in the VO, which is the old behaviour from before I introduced the conversion filter capability.
* hwtransfer: handle constraints for hwdec with NULL supported_formatsPhilip Langdale2023-10-221-3/+21
| | | | | | | | | | | | | | | | | | Some hwdecs (eg: dxva) have no frames constraints and no static supported_formats, which ends up segfaulting. This change fixes the segfault, but also includes additional changes to avoid concluding that direct output of hardware decoded video is impossible. In the case where there are no frame constraints and no supported_formats, we basically have no idea what the hardware actually supports. Previously, we just tried to display the frame, but all the work I did to detect incompatible formats causes this scenario to now conclude that no formats can be displayed, and forces a HW download to system memory. I have made a couple of small changes to assume that direct display is possible. If it's not, the VO will error out down the line, which is what happened previously.
* hwtransfer: handle hwcontexts that don't implement frame constraintsPhilip Langdale2023-10-161-3/+49
| | | | | | | | | | | | | | | Some ffmpeg hwcontexts do not implement frame constraints. That means they cannot report which formats are support for given configurations. My previous changes to make hwupload more capable relies on the constraints to be provided to accurately describe what combinations are supported, so we will currently see a bunch of errors failing to configure the hwupload filter on platforms such as v4l2 SoCs with drmprime. This doesn't break playback, but it's confusing. To bridge the gap, this change uses the static format metadata that we include for hwdecs to build a fake constraints struct to keep all the other code working.
* hwtransfer: make probe_formats logging less spammyllyyr2023-09-151-7/+26
| | | | | This demotes the logs from VERBOSE to DEBUG, as well as making it log formats in one line rather than one per line.
* hwtransfer: check if the source format is accepted directly by the VOPhilip Langdale2023-08-261-0/+26
| | | | | | | | | | | | | | | | | This may seem obvious in retrospect, but we need to explicitly account for the case where the source format is supported by the VO, but not a valid target format for the conversion filter. In that situation, we would conclude that conversion was necessary, because we relied solely on the conversion filter to identify acceptable target formats. To avoid that, we should go through the VO's reported set of supported formats and if the source format is on the list, ensure that format is also on the target list. This is mostly a no-op as most VOs do not report supported formats (instead assuming that all formats decoders can produce are supported) and in the case where it matters (vaapi), there is only one format that the VO supports which the conversion filter does not (yuv444p).
* hwtransfer: use the right hardware config to find conversion targetsPhilip Langdale2023-08-261-19/+59
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The last piece in the puzzle for doing hardware conversions automatically is ensuring we only consider valid target formats for the conversion. Although it is unintuitive, some vaapi drivers can expose a different set of formats for uploads vs for conversions, and that is the case on the Intel hardware I have here. Before this change, we would use the upload target list, and our selection algorithm would pick a format that doesn't work for conversions, causing everything to fail. Whoops. Successfully obtaining the conversion target format list is a bit of a convoluted process, with only parts of it encapsulated by ffmpeg. Specifically, ffmpeg understands the concept of hardware configurations that can affect the constraints of a device, but does not define what configurations are - that is left up to the specific hwdevice type. In the case of vaapi, we need to create a config for the video processing endpoint, and use that when querying for constraints. I decided to encapsulate creation of the config as part of the hwdec init process, so that the constraint query can be down in the hwtransfer code in an opaque way. I don't know if any other hardware will need this capability, but if so, we'll be able to account for it. Then, when we look at probing, instead of checking for what formats are supported for transfers, we use the results of the constraint query with the conversion config. And as that config doesn't depend on the source format, we only need to do it once.
* hwtransfer: implement support for hw->hw format conversionPhilip Langdale2023-08-261-35/+75
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Historically, we have not attempted to support hw->hw format conversion in the autoconvert logic. If a user needed to do these kinds of conversions, they needed to manually insert the hw format's conversion filter manually (eg: scale_vaapi). This was usually fine because the general rule is that any format supported by the hardware can be used as well as any other. ie: You would only need to do conversion if you have a specific goal in mind. However, we now have two situations where we can find ourselves with a hardware format being produced by a decoder that cannot be accepted by a VO via hwdec-interop: * dmabuf-wayland can only accept formats that the Wayland compositor accepts. In the case of GNOME, it can only accept a handful of RGB formats. * When decoding via VAAPI on Intel hardware, some of the more unusual video encodings (4:2:2, 10bit 4:4:4) lead to packed frame formats which gpu-next cannot handle, causing rendering to fail. In both these cases (at least when using VAAPI with dmabuf-wayland), if we could detect the failure case and insert a `scale_vaapi` filter, we could get successful output automatically. For `hwdec=drm`, there is currently no conversion filter, so dmabuf-wayland is still out of luck there. The basic approach to implementing this is to detect the case where we are evaluating a hardware format where the VO can accept the hardware format itself, but may not accept the underlying sw format. In the current code, we bypass autoconvert as soon as we see the hardware format is compatible. My first observation was that we actually have logic in autoconvert to detect when the input sw format is not in the list of allowed sw formats passed into the autoconverter. Unfortunately, we never populate this list, and the way you would expect to do that is vo-query-format returning sw format information, which it does not. We could define an extended vo-query-format-2, but we'd still need to implement the probing logic to fill it in. On the other hand, we already have the probing logic in the hwupload filter - and most recently I used that logic to implement conversion on upload. So if we could leverage that, we could both detect when hw->hw conversion is required, and pick the best target format. This exercise is then primarily one of detecting when we are in this case and letting that code run in a useful way. The hwupload filter is a bit awkward to work with today, and so I refactored a bunch of the set up code to actually make it more encapsulated. Now, instead of the caller instantiating it and then triggering the probe, we probe on creation and instantiate the correct underlying filter (hwupload vs conversion) automatically.
* f_hwtransfer: disable vulkan multiplane images when uploading from cudaPhilip Langdale2023-05-281-2/+3
| | | | | | | | | | | | | | | | Although we can support vulkan multiplane images, cuda lacks any such support, and so cannot natively import such images for interop. It's possible that we can do separate exports for each plane in the image and have it work, but for now, we can selectively disable multiplane when we know that we'll be consuming cuda frames. As a reminder, even though cuda is the frame source, interop is one way so the vulkan images have to be imported to cuda before we copy the frame contents over. This logic here is slightly more complex than I'd like but you can't just set the flag blindly, as it will cause hwframes ctx creation to fail if the format is packed or if it's planar rgb. Oh well.
* hwdec_vulkan: add Vulkan HW InteropPhilip Langdale2023-05-281-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Vulkan Video Decoding has finally become a reality, as it's now showing up in shipping drivers, and the ffmpeg support has been merged. With that in mind, this change introduces HW interop support for ffmpeg Vulkan frames. The implementation is functionally complete - it can display frames produced by hardware decoding, and it can work with ffmpeg vulkan filters. There are still various caveats due to gaps and bugs in drivers, so YMMV, as always. Primary testing has been done on Intel, AMD, and nvidia hardware on Linux with basic Windows testing on nvidia. Notable caveats: * Due to driver bugs, video decoding on nvidia does not work right now, unless you use the Vulkan Beta driver. It can be worked around, but requires ffmpeg changes that are not considered acceptable to merge. * Even if those work-arounds are applied, Vulkan filters will not work on video that was decoded by Vulkan, due to additional bugs in the nvidia drivers. The filters do work correctly on content decoded some other way, and then uploaded to Vulkan (eg: Decode with nvdec, upload with --vf=format=vulkan) * Vulkan filters can only be used with drivers that support VK_EXT_descriptor_buffer which doesn't include Intel ANV as yet. There is an MR outstanding for this. * When dealing with 1080p content, there may be some visual distortion in the bottom lines of frames due to chroma scaling incorporating the extra hidden lines at the bottom of the frame (1080p content is actually stored as 1088 lines), depending on the hardware/driver combination and the scaling algorithm. This cannot be easily addressed as the mechanical fix for it violates the Vulkan spec, and probably requires a spec change to resolve properly. All of these caveats will be fixed in either drivers or ffmpeg, and so will not require mpv changes (unless something unexpected happens) If you want to run on nvidia with the non-beta drivers, you can this ffmpeg tree with the work-around patches: * https://github.com/philipl/FFmpeg/tree/vulkan-nvidia-workarounds
* various: fix typosHarri Nieminen2023-03-281-1/+1
| | | | Found by codespell
* f_hwtransfer: fix leak of framesconstraintsThomas Weißschuh2023-02-031-0/+1
|
* filters/f_hwtransfer: remove VAAPI <-> Vulkan mapping for nowPhilip Langdale2022-10-291-4/+0
| | | | | | | | | This mapping isn't actually relevant until we have the Vulkan interop merged, and it requires a newer version of libavutil than our minimum requirement. So I'm going to remove it from master and put it in the interop PR. Fixes #10813
* vo_dmabuf_wayland: wayland VO displaying dmabuf buffersAaron Boxer2022-10-261-4/+4
| | | | | | | | | | | | | Wayland VO that can display images from either vaapi or drm hwdec The PR adds the following changes: 1. a context_wldmabuf context with no gl dependencies 2. no-op ra_wldmabuf and dmabuf_interop_wldmabuf objects no-op because there is no need to map/unmap the drmprime buffer, and there is no need to manage any textures. Tested on both x86_64 and rk3399 AArch64
* f_hwtransfer: allow hw uploads to implicitly convert formatsPhilip Langdale2022-10-151-26/+47
| | | | | | | | | | | | | | | | | | | | | | | | | | | | vaapi allows for implicit conversion on upload, which has some relevance as the set of supported source formats is larger than the set of displayable formats. In theory, this allows for offloading the conversion to the GPU - if you have any confidence in the hardware and/or driver's ability to do the conversion. Today, we actually track the 'input' and 'output' upload formats separately all the way up until the point we do a check as to whether the original source format is an accepted 'output' format and then reject it if it is not. This means that we're essentially ignoring all the work we did to track those 'input' formats in the first place. But it also means that it's a simple change to compare against the 'input' format instead. The logic is already in place to do best format selection on both sides. I imagine that if I read through the history here, wm4 tried to implement all of this properly and then gave up in disgust after seeing vaapi mangle various conversions. This is particularly interesting for vo-dmabuf-wayland where it is only possible to display the subset of valid vaapi formats that are supported by the compositor, yet all playback has to go through vaapi. Users will then be able to take advantage of all possible vaapi formats to avoid having to do software format conversion.
* f_hwtransfer: accept non-power-of-2 hw uploadsPhilip Langdale2022-10-151-5/+0
| | | | | This works fine for me on vaapi, cuda and vulkan. Why did we have this restriction in there? Worst case, the hwdec_interop will reject it.
* f_hwtransfer: get rid of the shit listPhilip Langdale2022-10-151-46/+0
| | | | | | | | | | | | | | | | | | | | | | | | | A few years ago, wm4 got sufficiently annoyed with how vaapi image format support was being discovered that he flipped the table and introduced the shit list (which just included vaapi) to hard-code the set of supported formats. While that might have been necessary at the time, I haven't been able to find a situation where the true list of supported formats was unsafe to use. We filter down the list based on what the vo reports - and the vo is already doing a thorough testing of formats, and if a format makes it through that gauntlet, it does actually work. Interestingly, as far as I can tell, the hwdec_vaapi probing code was already good enough at the time (also written by wm4), so perhaps the key difference here is that the driver side of things has improved. I dug into this because of the support for the 422/444 high bit depth vaapi formats I added to ffmpeg. These are obviously not in the hard coded list today, but they work fine. Finally, although it's positioned as a vaapi thing, it's really just Intel specific, as the AMD vaapi driver has never exposed support for anything except the formats used by the decoder/encoder profiles.
* f_hwtransfer: mp_image_pool: support HW -> HW mappingPhilip Langdale2022-09-211-1/+46
| | | | | | | | | | | | | | | | Certain combinations of hardware formats require the use of hwmap to transfer frames between the formats, rather than hwupload, which will fail if attempted. To keep the usage of vf_format for HW -> HW transfers as intuitive as possible, we should detect these cases and do the map operation instead of uploading. For now, the relevant cases are moving between VAAPI and Vulkan, and VAAPI and DRM Prime, in both directions. I have introduced the IMGFMT entry for Vulkan here so that I can put in the complete mapping table. It's actually not useless, as you can map to Vulkan, use a Vulkan filter and then map back to VAAPI for display output.
* f_autoconvert: f_hwtransfer: support HW -> HW uploadsPhilip Langdale2022-09-211-15/+50
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Historically, HW -> HW uploads did not exist, so the current code assumes they will never happen. But as part of introducing Vulkan support into ffmpeg, we added HW -> HW support to enable transfers between Vulkan and CUDA. Today, that means you can use the lavfi hwupload filter with the correct configuration (and previous changes in this series) but it would be more convenient to enable HW -> HW in the format filter so that the transfers can be done more intuitively: ``` --vf=format=fmt=cuda ``` and ``` --vf=format=fmt=vulkan ``` Most of the work here is skipping logic that is specific to SW -> HW uploads doing format conversion. There is no ability to do inline conversion when moving between HW formats, so the format must be mutually understood to begin with. Additional work needs to be done to enable transfers between VAAPI and Vulkan which uses mapping, rather than uploads. I'll tackle that in the next change.
* vo: hwdec: do hwdec interop lookup by image formatPhilip Langdale2022-09-211-2/+2
| | | | | | | | | | | | | | It turns out that it's generally more useful to look up hwdecs by image format, rather than device type. In the situations where we need to find one, we generally know the image format we're dealing with. Doing this avoids us having to create mappings from image format to device type. The most significant part of this change is filling in the image format for the various hw interops. There is a hw_imgfmt field today today, but only a couple of the interops fill it in, and that seems to be because we've never actually used this piece of metadata before. Well, now we have a good use for it.
* f_hwtransfer: extend vaapi whitelist with some working formatswm42020-01-171-1/+2
| | | | | | Notably, BGR0, which is the only additional format listed as supported by the texture mapper, results in broken colors. This is most likely not a mpv issue, so the whitelist fulfils its purpose.
* f_hwtransfer: minor debug logging improvementwm42020-01-171-2/+5
|
* f_hwtransfer: move format fields to private structwm42020-01-121-24/+37
| | | | | A user can barely do anything useful with it, and there are no users which access these anyway, so private is better.
* f_hwtransfer: restructure and error properly on broken caseswm42020-01-121-20/+24
| | | | | | | | | | | I think the previous code didn't consider the situation if the input format was not any of the upload formats. It then could have possibly tried to upload the wrong format (and not sure what the underlying APIs do with it). Take care of this, also improve logging, and change it such that mp_hwupload_find_upload_format() does not unnecessarily change the state (although it doesn't really matter).
* f_hwtransfer: slightly better loggingwm42020-01-111-3/+4
|
* f_hwtransfer: whitelist vaapi formats that actually appear to workwm42020-01-111-3/+50
| | | | | | | | | | | | | | | | | (Oh yes, we could have skipped all the complexity, and hardcoded the cases that work in the first place. This wouldn't be an issue if FFmpeg or libva exported correct information. Also possible that FFmpeg's filter chain does not allow to do this correctly in the first place, since filters expose next to no meta information about what hw formats etc. they need.) Note that uploading yuv420p to a nv12 vaapi surface actually works, but the blacklist excludes it. So this might get a bit slower. I'm not bothering with this case because it's rarely needed, and the blacklist specification would become a bit more complex if you had to specify sw/upload format pairs. Fixes: #7350
* f_hwtransfer: change order in which hwdec upload formats are consideredwm42020-01-111-26/+28
| | | | | | | | | | | | | | | | | | | | | Basically, instead of trusting the upload format, and picking the first sw format that has a desired upload format, trust the sw format. So now we pick the sw format first, and then select from the set of upload formats supported by it. This is probably more straightforward, and works around a crash with vaapi: mpv 8bit.mkv --vf=format=vaapi --gpu-hwdec-interop=all (Forces vaapi upload if hw decoding is not enabled.) Unfortunately, this still does not work, because vaapi, FFmpeg, the VO interop code, or all of them are doing something stupid. In particular, this picks the yuv420p sw format, which doesn't really exist despiter advertised (???????????????????????????????????????), and simply breaks. See: #7350
* f_hwtransfer: add a mp_hwdownload filterwm42019-10-021-0/+53
| | | | | | | | | This just wraps the mp_image_hw_download() function as a filter and adds some minor caching/error logging. (Shame that it needs to much boilerplate, I guess.) Will be used by the following commit. Wrapping it as filter seemed more convenient than other choices.
* f_hwtransfer: more detailed loggingwm42018-03-151-3/+4
| | | | This also switches the order, because that makes more sense.
* f_hwtransfer: fix a logic errorwm42018-03-151-2/+2
| | | | | Jesus Christ, how did I get this wrong, or never verified proper function. This fixes --vf=vdpaupp not working with yuv420p input.
* video: rewrite filtering glue codewm42018-01-301-0/+299
Get rid of the old vf.c code. Replace it with a generic filtering framework, which can potentially handle more than just --vf. At least reimplementing --af with this code is planned. This changes some --vf semantics (including runtime behavior and the "vf" command). The most important ones are listed in interface-changes. vf_convert.c is renamed to f_swscale.c. It is now an internal filter that can not be inserted by the user manually. f_lavfi.c is a refactor of player/lavfi.c. The latter will be removed once --lavfi-complex is reimplemented on top of f_lavfi.c. (which is conceptually easy, but a big mess due to the data flow changes). The existing filters are all changed heavily. The data flow of the new filter framework is different. Especially EOF handling changes - EOF is now a "frame" rather than a state, and must be passed through exactly once. Another major thing is that all filters must support dynamic format changes. The filter reconfig() function goes away. (This sounds complex, but since all filters need to handle EOF draining anyway, they can use the same code, and it removes the mess with reconfig() having to predict the output format, which completely breaks with libavfilter anyway.) In addition, there is no automatic format negotiation or conversion. libavfilter's primitive and insufficient API simply doesn't allow us to do this in a reasonable way. Instead, filters can use f_autoconvert as sub-filter, and tell it which formats they support. This filter will in turn add actual conversion filters, such as f_swscale, to perform necessary format changes. vf_vapoursynth.c uses the same basic principle of operation as before, but with worryingly different details in data flow. Still appears to work. The hardware deint filters (vf_vavpp.c, vf_d3d11vpp.c, vf_vdpaupp.c) are heavily changed. Fortunately, they all used refqueue.c, which is for sharing the data flow logic (especially for managing future/past surfaces and such). It turns out it can be used to factor out most of the data flow. Some of these filters accepted software input. Instead of having ad-hoc upload code in each filter, surface upload is now delegated to f_autoconvert, which can use f_hwupload to perform this. Exporting VO capabilities is still a big mess (mp_stream_info stuff). The D3D11 code drops the redundant image formats, and all code uses the hw_subfmt (sw_format in FFmpeg) instead. Although that too seems to be a big mess for now. f_async_queue is unused.