video: remove d3d11 video processor use from OpenGL interop

We now have a video filter that uses the d3d11 video processor, so it
makes no sense to have one in the VO interop code. The VO uses it for
formats not directly supported by ANGLE (so the video data is converted
to a RGB texture, which ANGLE can take in).

Change this so that the video filter is automatically inserted if
needed. Move the code that maps RGB surfaces to its own inteorp backend.
Add a bunch of new image formats, which are used to enforce the new
constraints, and to automatically insert the filter only when needed.

The added vf mechanism to auto-insert the d3d11vpp filter is very dumb
and primitive, and will work only for this specific purpose. The format
negotiation mechanism in the filter chain is generally not very pretty,
and mostly broken as well. (libavfilter has a different mechanism, and
these mechanisms don't match well, so vf_lavfi uses some sort of hack.
It only works because hwaccel and non-hwaccel formats are strictly
separated.)

The RGB interop is now only used with older ANGLE versions. The only
reason I'm keeping it is because it's relatively isolated (uses only
existing mechanisms and adds no new concepts), and because I want to be
able to compare the behavior of the old code with the new one for
testing. It will be removed eventually.

If ANGLE has NV12 interop, P010 is now handled by converting to NV12
with the video processor, instead of converting it to RGB and using the
old mechanism to import that as a texture.

1 files changed, 11 insertions, 1 deletions

diff --git a/video/img_format.h b/video/img_format.h
index 73c06787b1..b6f5830b8a 100644
--- a/video/img_format.h
+++ b/video/img_format.h
@@ -198,7 +198,17 @@ enum mp_imgfmt {
     IMGFMT_VDPAU,           // VdpVideoSurface
     IMGFMT_VDPAU_OUTPUT,    // VdpOutputSurface
     IMGFMT_VAAPI,
-    IMGFMT_D3D11VA,         // ID3D11VideoDecoderOutputView (NV12/P010/P016)
+    // NV12/P010/P016
+    // plane 1: ID3D11Texture2D
+    // plane 2: slice index casted to pointer
+    // plane 3: ID3D11VideoDecoderOutputView (can be absent in filters/VO)
+    IMGFMT_D3D11VA,
+    // Like IMGFMT_D3D11VA, but format is restricted to NV12.
+    IMGFMT_D3D11NV12,
+    // Like IMGFMT_D3D11VA, but format is restricted to a certain RGB format.
+    // Also, it must have a share handle, have been flushed, and not be a
+    // texture array slice.
+    IMGFMT_D3D11RGB,
     IMGFMT_DXVA2,           // IDirect3DSurface9 (NV12/P010/P016)
     IMGFMT_MMAL,            // MMAL_BUFFER_HEADER_T
     IMGFMT_VIDEOTOOLBOX,    // CVPixelBufferRef