options: fix OPT_BYTE_SIZE upper limits

As an unfortunate disaster, min/max values use the type double, which
causes tons of issues with int64_t types. Anyway, OPT_BYTE_SIZE is often
used as maximum for size_t quantities, which can have a size different
from (u)int64_t.

OPT_BYTE_SIZE still uses in64_t, because in theory, you could use it for
file sizes. (demux.c would for example be capable of caching more than
2GB on 32 bit platforms if a file cache is used. Though for some reason
the accounting code still uses size_t, so that use case is broken. But
still insist that it _could_ be used this way.)

There were various inconsistent attempts to set m_option.max to a value
such that the size_t/int64_t upper limit is not exceeded. Due to the
double max field, this didn't really work correctly. Try to fix this
with the M_MAX_MEM_BYTES constant. It's a good approximation, because on
32 bit it should allow 2GB (untested, also would probably exhaust
address space in practice but whatever), and something "high enough" in
64 bit.

For some reason, clang 11 still warns. But I think this might be a clang
bug, or I'm crazy. The result is correct anyway.

1 files changed, 2 insertions, 6 deletions

diff --git a/demux/demux.c b/demux/demux.c
index f5ced47887..478029b35a 100644
--- a/demux/demux.c
+++ b/demux/demux.c
@@ -106,8 +106,6 @@ struct demux_opts {
 
 #define OPT_BASE_STRUCT struct demux_opts
 
-#define MAX_BYTES MPMIN(INT64_MAX, SIZE_MAX / 2)
-
 static bool get_demux_sub_opts(int index, const struct m_sub_options **sub);
 
 const struct m_sub_options demux_conf = {
@@ -116,12 +114,10 @@ const struct m_sub_options demux_conf = {
             {"no", 0}, {"auto", -1}, {"yes", 1})},
         {"cache-on-disk", OPT_FLAG(disk_cache)},
         {"demuxer-readahead-secs", OPT_DOUBLE(min_secs), M_RANGE(0, DBL_MAX)},
-        // (The MAX_BYTES sizes may not be accurate because the max field is
-        // of double type.)
         {"demuxer-max-bytes", OPT_BYTE_SIZE(max_bytes),
-            M_RANGE(0, MAX_BYTES)},
+            M_RANGE(0, M_MAX_MEM_BYTES)},
         {"demuxer-max-back-bytes", OPT_BYTE_SIZE(max_bytes_bw),
-            M_RANGE(0, MAX_BYTES)},
+            M_RANGE(0, M_MAX_MEM_BYTES)},
         {"demuxer-donate-buffer", OPT_FLAG(donate_fw)},
         {"force-seekable", OPT_FLAG(force_seekable)},
         {"cache-secs", OPT_DOUBLE(min_secs_cache), M_RANGE(0, DBL_MAX),