vd_lavc: repeatedly attempt to fallback if hwdec fails in receive_frame

There is an additional failure path I didn't account for in my previous
work. While I ensured that a late hwdec failure in receive_frame can be
recovered from by trying the next hwdec, there is a specific
combination where if an hwdec fails in receive_frame, and the next
hwdec is a full decoder (eg: v4l2m2m), and that also fails, we are left
with no decoder and so the entire decoding process ends and playback is
stopped.

Basically, we must keep re-attempting the fallback in receive_frame
until we get a valid decoder (software or hardware). This edge case
will rarely be encountered as there are only a couple of decoder based
hwdecs.

Fixes #11947

1 files changed, 14 insertions, 2 deletions

diff --git a/video/decode/vd_lavc.c b/video/decode/vd_lavc.c
index 7260904b03..77f64aa8c5 100644
--- a/video/decode/vd_lavc.c
+++ b/video/decode/vd_lavc.c
@@ -1235,13 +1235,25 @@ static int receive_frame(struct mp_filter *vd, struct mp_frame *out_frame)
     int ret = decode_frame(vd);
 
     if (ctx->hwdec_failed) {
-        // Failed hardware decoding? Try again in software.
+        // Failed hardware decoding? Try the next one, and eventually software.
         struct demux_packet **pkts = ctx->sent_packets;
         int num_pkts = ctx->num_sent_packets;
         ctx->sent_packets = NULL;
         ctx->num_sent_packets = 0;
 
-        force_fallback(vd);
+        /*
+         * We repeatedly force_fallback until we get an avctx, because there are
+         * certain hwdecs that are really full decoders, and so if these fail,
+         * they also fail to give us a valid avctx, and the early return path
+         * here will simply give up on decoding completely if there is no
+         * decoder. We should never hit an infinite loop as the hwdec list is
+         * finite and we will eventually exhaust it and fall back to software
+         * decoding (and in practice, most hwdecs are hwaccels and so the
+         * decoder will successfully init even if the hwaccel fails later.)
+         */
+        do {
+            force_fallback(vd);
+        } while (!ctx->avctx);
 
         ctx->requeue_packets = pkts;
         ctx->num_requeue_packets = num_pkts;