vd_lavc: add gross workaround for nvdec/libavcodec broken API issue

libavcodec's nvdec wrapper can return invalid frames, that do not have
any data fields set. This is not allowed by the API, but why would they
follow their own API?

Add a workaround to specifically detect this situation. In practice,
this should fall back to software decoding if it happens too often in a
row. (But single errors are still tolerated, because I don't know why.)

Untested due to lack of hardware from the regrettable graphics company.

Better do this here than deal with the moronic project we unfortunately
depend on.

See: #7185

1 files changed, 10 insertions, 4 deletions

diff --git a/video/decode/vd_lavc.c b/video/decode/vd_lavc.c
index 2acfa9a9c0..3f0eedc47c 100644
--- a/video/decode/vd_lavc.c
+++ b/video/decode/vd_lavc.c
@@ -1055,15 +1055,21 @@ static int decode_frame(struct mp_filter *vd)
     // data.
     assert(ctx->pic->buf[0]);
 
-    ctx->hwdec_fail_count = 0;
-
     struct mp_image *mpi = mp_image_from_av_frame(ctx->pic);
     if (!mpi) {
         av_frame_unref(ctx->pic);
         return ret;
     }
-    if (mpi->imgfmt == IMGFMT_CUDA)
-        assert(mpi->planes[0]);
+
+    if (mpi->imgfmt == IMGFMT_CUDA && !mpi->planes[0]) {
+        MP_ERR(vd, "CUDA frame without data. This is a FFmpeg bug.\n");
+        talloc_free(mpi);
+        handle_err(vd);
+        return AVERROR_BUG;
+    }
+
+    ctx->hwdec_fail_count = 0;
+
     mpi->pts = mp_pts_from_av(ctx->pic->pts, &ctx->codec_timebase);
     mpi->dts = mp_pts_from_av(ctx->pic->pkt_dts, &ctx->codec_timebase);