render: avoid overflow on BorderStyle=4 img allocation

The buffer was allocated and memset using the unchecked result of
multiplying the width and height of the image. This multiplication can
overflow in two ways, both of which ultimately lead to allocating a too
small buffer and later OOB reads when the API user processes the image.
Nothing in libass processes the image buffer after this initial alloc
and memset.

Width and height are ints and capped to the extent of the render frame.
ass_aligned_alloc and memeset use size_t for the the size parameters.

For one, since both operands are sigend integers an overflow in the
multiplication is undefined behaviour. In practice on common systems
today, it will wrap around and then through conversion to size_t
effectively limited to UINT_MAX.

However while uncommon, SIZE_MAX may also be smaller than either
UINT_MAX or INT_MAX. As a result the multiplication itself might not
overflow, but the conversion to size_t will truncate the result.
(ISO C99 only requires SIZE_MAX to be >= 65535 with
 no restrictions relative to other types.)

To avoid both error sources omit the BorderStyle=4 background box
if the image’s pixel count exceeds the limits of int or size_t.

The first problem was discovered by CodeQL.

1 files changed, 1 insertions, 1 deletions

diff --git a/libass/ass_render.c b/libass/ass_render.c
index f95479c..138c11a 100644
--- a/libass/ass_render.c
+++ b/libass/ass_render.c
@@ -2812,7 +2812,7 @@ static void add_background(RenderContext *state, EventImages *event_images)
     bottom      = FFMINMAX(bottom, 0, render_priv->height);
     int w = right - left;
     int h = bottom - top;
-    if (w < 1 || h < 1)
+    if (w < 1 || h < 1 || w > FFMIN(INT_MAX, SIZE_MAX) / h)
         return;
     void *nbuffer = ass_aligned_alloc(1, w * h, false);
     if (!nbuffer)