vdpau: split off decoder parts, use "new" libavcodec vdpau hwaccel API

Move the decoder parts from vo_vdpau.c to a new file vdpau_old.c. This
file is named so because because it's written against the "old"
libavcodec vdpau pseudo-decoder (e.g. "h264_vdpau").

Add support for the "new" libavcodec vdpau support. This was recently
added and replaces the "old" vdpau parts. (In fact, Libav is about to
deprecate and remove the "old" API without deprecation grace period,
so we have to support it now. Moreover, there will probably be no Libav
release which supports both, so the transition is even less smooth than
we could hope, and we have to support both the old and new API.)

Whether the old or new API is used is checked by a configure test: if
the new API is found, it is used, otherwise the old API is assumed.

Some details might be handled differently. Especially display preemption
is a bit problematic with the "new" libavcodec vdpau support: it wants
to keep a pointer to a specific vdpau API function (which can be driver
specific, because preemption might switch drivers). Also, surface IDs
are now directly stored in AVFrames (and mp_images), so they can't be
forced to VDP_INVALID_HANDLE on preemption. (This changes even with
older libavcodec versions, because mp_image always uses the newer
representation to make vo_vdpau.c simpler.)

Decoder initialization in the new code tries to deal with codec
profiles, while the old code always uses the highest profile per codec.

Surface allocation changes. Since the decoder won't call config() in
vo_vdpau.c on video size change anymore, we allow allocating surfaces
of arbitrary size instead of locking it to what the VO was configured.
The non-hwdec code also has slightly different allocation behavior now.

Enabling the old vdpau special decoders via e.g. --vd=lavc:h264_vdpau
doesn't work anymore (a warning suggesting the --hwdec option is
printed instead).

2 files changed, 0 insertions, 6 deletions

diff --git a/video/filter/vf.h b/video/filter/vf.h
index d115890f64..c792049b3b 100644
--- a/video/filter/vf.h
+++ b/video/filter/vf.h
@@ -99,8 +99,6 @@ typedef struct vf_seteq {
 #define VFCTRL_SET_PP_LEVEL 5       // set postprocessing level
 #define VFCTRL_SET_EQUALIZER 6 // set color options (brightness,contrast etc)
 #define VFCTRL_GET_EQUALIZER 8 // get color options (brightness,contrast etc)
-#define VFCTRL_HWDEC_DECODER_RENDER 9 // vdpau hw decoding
-#define VFCTRL_HWDEC_ALLOC_SURFACE 10 // vdpau hw decoding
 #define VFCTRL_SCREENSHOT      14  // Take screenshot, arg is voctrl_screenshot_args
 #define VFCTRL_INIT_OSD        15  // Filter OSD renderer present?
 #define VFCTRL_SET_DEINTERLACE 18  // Set deinterlacing status
diff --git a/video/filter/vf_vo.c b/video/filter/vf_vo.c
index 60113192b7..9edb393e23 100644
--- a/video/filter/vf_vo.c
+++ b/video/filter/vf_vo.c
@@ -87,10 +87,6 @@ static int control(struct vf_instance *vf, int request, void *data)
         };
         return vo_control(video_out, VOCTRL_GET_EQUALIZER, &param) == VO_TRUE;
     }
-    case VFCTRL_HWDEC_DECODER_RENDER:
-        return vo_control(video_out, VOCTRL_HWDEC_DECODER_RENDER, data);
-    case VFCTRL_HWDEC_ALLOC_SURFACE:
-        return vo_control(video_out, VOCTRL_HWDEC_ALLOC_SURFACE, data);
     }
     return CONTROL_UNKNOWN;
 }