vd_lavc: uninit the hwdec backend after closing the decoder

A recent behavior change in libavcodec's h264 decoder keeps at least 1
surface even after avcodec_flush_buffers() has been called. We used to
flush the decoder in order to make sure all surfaces are free'd, so that
the hw decoder can be safely uninitialized. This doesn't work anymore.

Fix it by closing the AVCodecContext before the hw decoder is
uninitialized. This is actually simpler and more robust. It seems to be
well-supported too.

Fixes invalid read accesses with vaapi-copy and dxva2-copy. These
destroyed the hwdec API fully on uninit, and could not deal with
surfaces surviving the decoder.

Probably fixes #1587.

(cherry picked from commit cf073138b289243fb551242f8058a4f8490cc9af)

1 files changed, 3 insertions, 6 deletions

diff --git a/video/decode/vd_lavc.c b/video/decode/vd_lavc.c
index 109099e382..1982656f75 100644
--- a/video/decode/vd_lavc.c
+++ b/video/decode/vd_lavc.c
@@ -436,12 +436,6 @@ static void uninit_avctx(struct dec_video *vd)
     vd_ffmpeg_ctx *ctx = vd->priv;
     AVCodecContext *avctx = ctx->avctx;
 
-    if (avctx && avcodec_is_open(avctx))
-        avcodec_flush_buffers(avctx);
-
-    if (ctx->hwdec && ctx->hwdec->uninit)
-        ctx->hwdec->uninit(ctx);
-
     if (avctx) {
         if (avctx->codec && avcodec_close(avctx) < 0)
             MP_ERR(vd, "Could not close codec.\n");
@@ -450,6 +444,9 @@ static void uninit_avctx(struct dec_video *vd)
         av_freep(&avctx->slice_offset);
     }
 
+    if (ctx->hwdec && ctx->hwdec->uninit)
+        ctx->hwdec->uninit(ctx);
+
     av_freep(&ctx->avctx);
 
     av_frame_free(&ctx->pic);