vdpau: split off decoder parts, use "new" libavcodec vdpau hwaccel API

Move the decoder parts from vo_vdpau.c to a new file vdpau_old.c. This
file is named so because because it's written against the "old"
libavcodec vdpau pseudo-decoder (e.g. "h264_vdpau").

Add support for the "new" libavcodec vdpau support. This was recently
added and replaces the "old" vdpau parts. (In fact, Libav is about to
deprecate and remove the "old" API without deprecation grace period,
so we have to support it now. Moreover, there will probably be no Libav
release which supports both, so the transition is even less smooth than
we could hope, and we have to support both the old and new API.)

Whether the old or new API is used is checked by a configure test: if
the new API is found, it is used, otherwise the old API is assumed.

Some details might be handled differently. Especially display preemption
is a bit problematic with the "new" libavcodec vdpau support: it wants
to keep a pointer to a specific vdpau API function (which can be driver
specific, because preemption might switch drivers). Also, surface IDs
are now directly stored in AVFrames (and mp_images), so they can't be
forced to VDP_INVALID_HANDLE on preemption. (This changes even with
older libavcodec versions, because mp_image always uses the newer
representation to make vo_vdpau.c simpler.)

Decoder initialization in the new code tries to deal with codec
profiles, while the old code always uses the highest profile per codec.

Surface allocation changes. Since the decoder won't call config() in
vo_vdpau.c on video size change anymore, we allow allocating surfaces
of arbitrary size instead of locking it to what the VO was configured.
The non-hwdec code also has slightly different allocation behavior now.

Enabling the old vdpau special decoders via e.g. --vd=lavc:h264_vdpau
doesn't work anymore (a warning suggesting the --hwdec option is
printed instead).

1 files changed, 18 insertions, 1 deletions

diff --git a/video/decode/lavc.h b/video/decode/lavc.h
index 25ed2b8ac5..3611530400 100644
--- a/video/decode/lavc.h
+++ b/video/decode/lavc.h
@@ -10,7 +10,7 @@
 #include "demux/stheader.h"
 #include "video/mp_image.h"
 
-typedef struct ffmpeg_ctx {
+typedef struct lavc_ctx {
     AVCodecContext *avctx;
     AVFrame *pic;
     struct hwdec *hwdec;
@@ -23,11 +23,28 @@ typedef struct ffmpeg_ctx {
     enum AVDiscard skip_frame;
     const char *software_fallback_decoder;
 
+    // From VO
+    struct mp_hwdec_info *hwdec_info;
+
+    // For free use by hwdec implementation
+    void *hwdec_priv;
+
+    // Legacy
     bool do_dr1;
     struct FramePool *dr1_buffer_pool;
     struct mp_image_pool *non_dr1_pool;
 } vd_ffmpeg_ctx;
 
+struct vd_lavc_hwdec_functions {
+    // If not-NULL, a 0 terminated list of IMGFMT_ formats. Only one of these
+    // formats is accepted when handling the libavcodec get_format callback.
+    const int *image_formats;
+    int (*init)(struct lavc_ctx *ctx);
+    void (*uninit)(struct lavc_ctx *ctx);
+    struct mp_image *(*allocate_image)(struct lavc_ctx *ctx, AVFrame *frame);
+    void (*fix_image)(struct lavc_ctx *ctx, struct mp_image *img);
+};
+
 // lavc_dr1.c
 int mp_codec_get_buffer(AVCodecContext *s, AVFrame *frame);
 void mp_codec_release_buffer(AVCodecContext *s, AVFrame *frame);