demux: add sanity checks to packet allocation functions

Change new_demux_packet() and resize_demux_packet() length parameter
type from int to size_t and add a check to abort() if the size is over
1 GB. This should make integer overflow problems leading to memory
corruption in demuxers less likely; and aborting should be no worse
than insane memory consumption. Also make the functions abort() if the
actual allocation fails instead of trying to continue with a
zero-sized buffer.

1 files changed, 2 insertions, 2 deletions

diff --git a/libmpdemux/demuxer.h b/libmpdemux/demuxer.h
index 3a7a6157c2..742e42aa74 100644
--- a/libmpdemux/demuxer.h
+++ b/libmpdemux/demuxer.h
@@ -286,8 +286,8 @@ typedef struct {
   int aid, vid, sid; //audio, video and subtitle id
 } demux_program_t;
 
-struct demux_packet *new_demux_packet(int len);
-void resize_demux_packet(struct demux_packet *dp, int len);
+struct demux_packet *new_demux_packet(size_t len);
+void resize_demux_packet(struct demux_packet *dp, size_t len);
 struct demux_packet *clone_demux_packet(struct demux_packet *pack);
 void free_demux_packet(struct demux_packet *dp);