sub: fix crash with certain uses of --vf=sub

If, for some reason, the subtitle renderer attempts to render a
subtitle before SD_CTRL_SET_VIDEO_PARAMS was called, it passed a
value calculated from invalid values. This can happen with --vf=sub
and --start. The crash happens if 1. there was a subtitle packet that
falls into the timestamp of the rendered video frame, 2. the playloop
hasn't informed the subtitle decoder about the video resolution yet
(normally unneeded, because that is used for weird corner cases only,
so this code is a bit fuzzy), and 3. something actually requests a
frame to be drawn from the subtitle renderer, like with vf_sub.

The actual crash was due to passing NaN as pixel aspect to libass,
which then created glyphs with ridiculous sizes, involving a few
integer overflows and unchecked mallocs.

The sd_lavc.c and sd_spu.c cases probably don't crash, but I'm not
sure, and it's better fix them anyway.

Not bothering with sd_spu.c, this crap is for compatibility and will
be removed soon.

Note that this would have been no problem, had the code checked whether
SD_CTRL_SET_VIDEO_PARAMS was actually called. This commit adds such a
check (although it basically checks after using the parameters).

Regression since 49caa0a7 and 633fde4a.

1 files changed, 4 insertions, 1 deletions

diff --git a/sub/sd_ass.c b/sub/sd_ass.c
index 4b0c3024c8..8b6f1f1554 100644
--- a/sub/sd_ass.c
+++ b/sub/sd_ass.c
@@ -19,6 +19,7 @@
 #include <stdlib.h>
 #include <assert.h>
 #include <string.h>
+#include <math.h>
 
 #include <libavutil/common.h>
 #include <ass/ass.h>
@@ -142,9 +143,11 @@ static void get_bitmaps(struct sd *sd, struct mp_osd_res dim, double pts,
                                opts->ass_vsfilter_aspect_compat))
     {
         // Let's use the original video PAR for vsfilter compatibility:
-        scale = scale
+        double par = scale
             * (ctx->video_params.d_w / (double)ctx->video_params.d_h)
             / (ctx->video_params.w   / (double)ctx->video_params.h);
+        if (isnormal(par))
+            scale = par;
     }
     mp_ass_configure(renderer, opts, &dim);
     ass_set_aspect_ratio(renderer, scale, 1);