vdpau: split off decoder parts, use "new" libavcodec vdpau hwaccel API

Move the decoder parts from vo_vdpau.c to a new file vdpau_old.c. This
file is named so because because it's written against the "old"
libavcodec vdpau pseudo-decoder (e.g. "h264_vdpau").

Add support for the "new" libavcodec vdpau support. This was recently
added and replaces the "old" vdpau parts. (In fact, Libav is about to
deprecate and remove the "old" API without deprecation grace period,
so we have to support it now. Moreover, there will probably be no Libav
release which supports both, so the transition is even less smooth than
we could hope, and we have to support both the old and new API.)

Whether the old or new API is used is checked by a configure test: if
the new API is found, it is used, otherwise the old API is assumed.

Some details might be handled differently. Especially display preemption
is a bit problematic with the "new" libavcodec vdpau support: it wants
to keep a pointer to a specific vdpau API function (which can be driver
specific, because preemption might switch drivers). Also, surface IDs
are now directly stored in AVFrames (and mp_images), so they can't be
forced to VDP_INVALID_HANDLE on preemption. (This changes even with
older libavcodec versions, because mp_image always uses the newer
representation to make vo_vdpau.c simpler.)

Decoder initialization in the new code tries to deal with codec
profiles, while the old code always uses the highest profile per codec.

Surface allocation changes. Since the decoder won't call config() in
vo_vdpau.c on video size change anymore, we allow allocating surfaces
of arbitrary size instead of locking it to what the VO was configured.
The non-hwdec code also has slightly different allocation behavior now.

Enabling the old vdpau special decoders via e.g. --vd=lavc:h264_vdpau
doesn't work anymore (a warning suggesting the --hwdec option is
printed instead).

1 files changed, 73 insertions, 0 deletions

diff --git a/video/vdpau.c b/video/vdpau.c
new file mode 100644
index 0000000000..697bda8447
--- /dev/null
+++ b/video/vdpau.c
@@ -0,0 +1,73 @@
+/*
+ * This file is part of mpv.
+ *
+ * mpv is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * mpv is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with mpv.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include "vdpau.h"
+
+#include "video/img_format.h"
+
+// Check whether vdpau initialization and preemption status is ok and we can
+// proceed normally.
+bool mp_vdpau_status_ok(struct mp_vdpau_ctx *ctx)
+{
+    return ctx->status_ok(ctx);
+}
+
+struct mp_image *mp_vdpau_get_video_surface(struct mp_vdpau_ctx *ctx, int fmt,
+                                            VdpChromaType chroma, int w, int h)
+{
+    return ctx->get_video_surface(ctx, fmt, chroma, w, h);
+}
+
+bool mp_vdpau_get_format(int imgfmt, VdpChromaType *out_chroma_type,
+                         VdpYCbCrFormat *out_pixel_format)
+{
+    VdpChromaType chroma = VDP_CHROMA_TYPE_420;
+    VdpYCbCrFormat ycbcr = (VdpYCbCrFormat)-1;
+
+    switch (imgfmt) {
+    case IMGFMT_420P:
+        ycbcr = VDP_YCBCR_FORMAT_YV12;
+        break;
+    case IMGFMT_NV12:
+        ycbcr = VDP_YCBCR_FORMAT_NV12;
+        break;
+    case IMGFMT_YUYV:
+        ycbcr = VDP_YCBCR_FORMAT_YUYV;
+        chroma = VDP_CHROMA_TYPE_422;
+        break;
+    case IMGFMT_UYVY:
+        ycbcr = VDP_YCBCR_FORMAT_UYVY;
+        chroma = VDP_CHROMA_TYPE_422;
+        break;
+    case IMGFMT_VDPAU_MPEG1:
+    case IMGFMT_VDPAU_MPEG2:
+    case IMGFMT_VDPAU_H264:
+    case IMGFMT_VDPAU_WMV3:
+    case IMGFMT_VDPAU_VC1:
+    case IMGFMT_VDPAU_MPEG4:
+    case IMGFMT_VDPAU:
+        break;
+    default:
+        return false;
+    }
+
+    if (out_chroma_type)
+        *out_chroma_type = chroma;
+    if (out_pixel_format)
+        *out_pixel_format = ycbcr;
+    return true;
+}