summaryrefslogtreecommitdiffstats
path: root/demux
Commit message (Collapse)AuthorAgeFilesLines
* demux, f_decoder_wrapper: fix coverart in backward modewm42019-09-191-11/+11
| | | | | | | | | | | | Shitty ancient hack that wastes my time all the time. demux.c: always return the coverart packet as soon as possible, and don't let the backward demux state machine possibly stop it. f_decoder_wrapper.c: mess with some shit until it somehow starts to work. I think the old code tried to let it cleverly fall through so the packet was processed "normally"; just make it run the "usual" code instead.
* demux_lavf: implement bad hack for backward playback of wavwm42019-09-191-4/+64
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit generally fixes backward playing in wav, at least in most PCM cases. libavformat's wav demuxer (and actually all other raw PCM based demuxers) have a specific behavior that breaks backward demuxing. The same thing also breaks persistent seek ranges in the demuxer cache, although that's less critical (it just means some cached data gets discarded). The backward demuxing issue is fatal, will log the message "Demuxer not cooperating.", and then typically stop doing anything. Unlike modern media formats, these formats don't organize media data in packets, but just wrap a monolithic byte stream that is described by a header. This is good enough for PCM, which uses fixed frames (a single sample for all audio channels), and for which it would be too expensive to have per frame headers. libavformat (and mpv) is heavily packet based, and using a single packet for each PCM frame causes too much overhead. So they typically "bundle" multiple frames into a single packet. This packet size is obviously arbitrary, and in libavformat's case hardcoded in its source code. The problem is that seeking doesn't respect this arbitrary packet boundary. Seeking is sample accurate. You can essentially seek inside a packet. The resulting packets will not be aligned with previously demuxed packets. This is normally OK. Backward seeking (and some other demuxer layer features) expect that demuxing an earlier demuxed file position eventually results in the same packets, regardless of the seeks that were done to get there. I like to call this "deterministic" demuxing. Backward demuxing in particular requires this to avoid overlaps, which would make it rather hard to get continuous output. Fix this issue by detecting wav and hopefully other raw audio formats with a heuristic (even PCM needs to be detected as heuristic). Then, if a seek is requested, align the seek timestamps on the guessed number of samples in the audio packets returned by the demuxer. The heuristic excludes files with multiple streams. (Except "attachment" video streams, which could be an ID3 tag. Yes, FFmpeg allows ID3 tags on WAV files.) Such files will inherently use the packet concept in some way. We don't know how the demuxer chooses the internal packet size, but we assume that it's fixed and aligned to PCM frame sizes. The frame size is most likely given by block_align (the native wav frame size, according to Microsoft). We possibly need to explicitly read and discard a packet if the seek is done without reading anything before that. We ignore any subsequent packet sizes; we need to avoid the very last packet, which likely has a different size. This hack should be rather benign. In the worst case, it will "round" the seek target a little, but the maximum rounding amount is bounded. Maybe we _could_ round up if SEEK_FORWARD is specified, but I didn't bother. An earlier commit fixed the same issue for mpv's demux_raw. An alternative, and probably much better solution would be clipping decoded data by timestamp. demux.c could allow the type of overlap the wav demuxer introduces, and instruct the decoder to clip the output against the last decoded timestamp. There's already an infrastructure for this (demux_packet.end field) used by EDL/ordered chapters. Although this sounds like a good solution, mpv unfortunately uses floats for timestamps. The rounding errors break sample accuracy. Even if you used integers, you'd need a timebase that is sample accurate (not always easy, since EDL can merge tracks with different sample rates).
* demux: add an explicit start state for backward demuxingwm42019-09-191-20/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Yay, more subtle state on top of this nightmarish, fragile state machine. But this is what happens when you subvert the laws of nature. This simple checks where playback should "resume" from when no packets were returned to the decoder yet after the seek that initiated backward playback. The main purpose is to process the first returned keyframe range in the same way like all other ranges. This ensures that things like preroll are included properly. Before this commit, it could for example have happened that the start of the first audio frame was slightly broken, because no preroll was included. Since the audio frame is reversed before sending it to the audio output, it would have added an audible discontinuity before the second frame was played; all subsequent frames would have been fine. (Although I didn't test and confirm this particular issue.) In future, this could be useful for certain other things. At least the condition for delaying the backstep seek becomes simpler and more explicit. Move the code that attempts to start demuxing up in dequeue_packet. Before, it was not called when the stream was in back_restarting state. This commit makes streams be in back_restarting state at initialization, so the demuxer would never have started reading. Likewise, we need to call back_demux_see_packets() right after seek in case the seek was within the cache. (We don't bother with checking whether it was a cached seek; nothing happens if it was a normal one.) There is nothing else that would process these cached packets explicitly, although coincidences could sporadically trigger it. The check for back_restart_next in find_backward_restart_pos() now decides whether to use this EOF special code. Since the backward playback start state also sets this variable, we don't need some of the complex checks in dequeue_packet() anymore either.
* demux: add a special case for backward demuxing of opuswm42019-09-191-1/+4
| | | | | | | | | | | Make --audio-backward-overlap default to 2 for Opus. I have no idea why this is needed. It seems to fix backward decoding though (going purely by listening). Normally, this should not be needed, since initial padding is completely contained within the first packet (normally, and in the case I tested). So the 2nd packet/frame should be fine, but for some unknown reason it works only with the 3rd.
* demux: use no overlapping packets for lossless audiowm42019-09-191-2/+3
| | | | | | Worthless optimization, but at least it justifies that the --audio-backward-overlap option has an "auto" choice. Tested with PCM and FLAC.
* demux: remove some redundant pointer indirectionswm42019-09-191-13/+13
| | | | | | In all of these cases ds->in should be the same as the local variable in, and neither ds->in nor in ever change, i.e. a cosmetic simplification.
* demux: change backward-overlap to keyframe ranges instead of packetswm42019-09-191-34/+36
| | | | | | | | | This seems more useful in general. This change also happens to fix a miscounting of preroll packets when some of them were "rounded" away, and which could make it stuck. Also a simple intra-refresh encode with x264 (and muxed to mkv by it) seems to work now. I guess I misinterpreted earlier results.
* demux: fix typoswm42019-09-191-2/+2
|
* demux: redo backstep seek handling slightly againwm42019-09-191-2/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Backstepping still could get "stuck" if the demuxer didn't seek far back enough. This commit fixes getting stuck if playing backwards from the end, and audio has ended much earlier than the video. In commit "demux: fix initial backward demuxing state in some cases", I claimed that the backward seek semantics ("snapping" backward in normal seeking, unrelated to backward playing) would take care of this. Unfortunately, this is not always quite true. In theory, a seek to any position (that does not use SEEK_FORWARD, i.e. backward snapping) should return a packet for every stream. But I have a mkv sample, where audio ends much earlier than video. Its mkvmerge created index does not have entries for audio packets, so the video index is used. This index has its last entry somewhere close after the end of audio. So no audio packets will be returned. With a "too small" back_seek_size, the demuxer will retry a seek target that ends up in this place forever. (This does not happen if you use --index=recreate. It also doesn't happen with libavformat, which always prefers its own index, while mpv's internal mkv demuxer strictly prefers the index from the file if it can be read.) Fix this by adding the back_seek_size every time we fail to see enough packets. This way the seek step can add up until it works. To prevent that back_seek_pos just "runs away" towards negative infinity by subtracting back_seek_size every time we back step to undo forward reading (e.g. if --no-cache is used), readjust the back_seek_pos to the lowest known resume position. (If the cache is active, kf_seek_pts can be used, but to work in all situations, the code needs to grab the minimum PTS in the keyframe range.)
* demux: set SEEK_HR for backstep seeks, move a hr-seek detail to playloopwm42019-09-191-10/+1
| | | | | | Just rearranging shit. Setting SEEK_HR for backstep seeks actually doesn't have much meaning, but disables the weird audio snapping for "keyframe" seeks, and I don't know it's late.
* demux: rename a variablewm42019-09-191-8/+8
| | | | | It's "better". This is all what's left from an attempt to make the code slightly nicer.
* demux: remove minor code duplicationwm42019-09-191-16/+14
| | | | | | | | | | | | | | | | | | This code used to be simpler, but now it's enough that it should be factored into a single function. Both uses of the new function are annoyingly different. The first use is the special case when a decoder tries to read packets, but the demuxer doesn't see any (like mp4 files with sparse video packets, which actually turned out to be chapter thumbnail "tracks"). Then the other stream queues will overflow, and the stream with no packets is marked EOF to avoid stalling playback. The second case is when the demxuer returns global EOF. It would be more awkward to have the loop iterating the streams in the function, because then you'd need a weird parameter to control the behavior.
* demux: fix initial backward demuxing state in some caseswm42019-09-191-59/+95
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Just "mpv file.mkv --play-direction=backward" did not work, because backward demuxing from the very end was not implemented. This is another corner case, because the resume mechanism so far requires a packet "position" (dts or pos) as reference. Now "EOF" is another possible reference. Also, the backstep mechanism could cause streams to find different playback start positions, basically leading to random playback start (instead of what you specified with --start). This happens only if backstep seeks are involved (i.e. no cached data yet), but since this is usually the case at playback start, it always happened. It was racy too, because it depended on the order the decoders on other threads requested new data. The comment below "resume_earlier" has some more blabla. Some other details are changed. I'm giving up on the "from_cache" parameter, and don't try to detect the situation when the demuxer does not seek properly. Instead, always seek back, hopefully some more. Instead of trying to adjust the backstep seek target by a random value of 1.0 seconds. Instead, always rely on the random value provided by the user via --demuxer-backward-playback-step. If the demuxer should really get "stuck" and somehow miss the seek target badly, or the user sets the option value to 0, then the demuxer will not make any progress and just eat CPU. (Although due to backward seek semantics used for backstep seeks, even a very small seek step size will work. Just not 0.) It seems this also fixes backstepping correctly when the initial seek ended at the last keyframe range. (The explanation above was about the case when it ends at EOF. These two cases are different. In the former, you just need to step to the previous keyframe range, which was broken because it didn't always react correctly to reaching EOF. In the latter, you need to do a separate search for the last keyframe.)
* demux_lavf: also fix cache seeking with large codec delaywm42019-09-191-0/+2
| | | | | | | | | Fixes the same thing as the previous commit did with demux_mkv. I'm not sure if this is correct or a good idea (well, it works with my sample file). There are some shady things in this, but describing them would require too many expletives.
* demux, demux_mkv: fix seeking in cache with large codec delaywm42019-09-193-1/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | In this scenario, the demuxer will output timestamps offset by the codec delay (e.g. negative timestamps at the start; mkv simulates those), and the trimming in the decoder (often libavcodec, but ad_lavc.c in our case) will adjust the timestamps back (e.g. stream actually starts at 0). This offset needs to be taken into account when seeking. This worked in the uncached case. (demux_mkv.c is a bit tricky in that the index is already in the offset space, so it compensates even though the seek call does not reference codec_delay.) But in the cached case, seeks backwards did not seek enough, and forward they seeked too much. Fix this by adding the codec delay to the index search. We need to get "earlier" packets, so e.g. seeking to position 0 really gets the initial packets with negative timestamps. This also adjusts the seek range start. This is also pretty obvious: if the beginning of the file is cached, the seek range should start at 0, not a negative value. We compare 0-based timestamps to it later on. Not sure if this is the best approach. I also could have thought about/checked some corner cases harder. But fuck this shit. Not fixing duration (who cares) or end trimming, which would reduce the seek range and duration (who cares).
* demux_mkv: stop setting per-packet initial padding from codec delaywm42019-09-191-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | This is a bad approach, and should be handled by a codec parameter field (in mp_codec_params or AVCodecParameters). It's bad because it's overly complicated, and has potential to break demuxer cache assumptions: packets that were "intended" for seek resuming may suddenly appear in the middle of a stream, when you seek back and play a cached part again. (In general it was fine though, because seek range joining tends to remove the first audio packet of the next range when trying to find an overlap.) demux_mkv.c does not try to export its codec_delay field through the codec parameters mentioned above. In the only case I spotted this element, the codec itself (opus) set this field within libavcodec. And I think that's actually how it should be. On the other hand, a file could in theory set this field via mkv headers if the codec is too stupid to have such a field internally. But I don't really care until I see such a file. The end trimming is still sort of needed (though not sure if anything uses it, other than the opus/mkv test sample I was using). The decoder can't know whether something is the last packet, until it's too late. The codec_delay field is still needed to offset timestamps.
* demux: don't adjust internal backward playback seeks by start timewm42019-09-191-3/+5
| | | | | | Only timestamps that enter or leave the demuxer API should be adjusted by ts_offset (which is usually the start time). queue_seek() is also used by backward demux seeks, which uses an internal timestamp.
* demux_raw: fix operation with demuxer cache and backward playbackwm42019-09-191-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Raw audio formats can be accessed sample-wise, and logically audio packets demuxed from it would contain only 1 sample. This is inefficient, so raw audio demuxers typically "bundle" multiple samples in one packet. The problem for the demuxer cache and backward playback is that they need properly aligned packets to make seeking "deterministic". The requirement is that if you read some packets, and then seek back, you eventually see the same packets again. demux_raw basically allowed to seek into the middle of a previously returned packet, which makes it impossible to make the transition seamless. (Unless you'd be aware of the packet data format and cut them to make it seamless, which is too complex for such a use case.) Solve this by always aligning seeks to packet boundaries. This reduces the seek accuracy to the arbitrarily chosen packet size. But you can use hr-seek to fix this. The gain from not making raw audio an awful special case pays in exchange for this "stupid" suggestion to use hr-seek. It appears this also fixes that it could and did seek into the middle of the frame (not sure if this code was ever tested - it goes back to removing the code duplication between the former demux_rawaudio.c and demux_rawvideo.c). If you really cared, you could introduce a seek flag that controls whether the seek is aligned or not. Then code which requires "deterministic" demuxing could set it. But this isn't really useful for us, and we'd always set the flag anyway, unless maybe the caching were forced disabled. libavformat's wav demuxer exhibits the same issue. We can't fix it (it would require the unpleasant experience of contributing to FFmpeg), so document this in otions.rst. In theory, this also affects seek range joining, but the only bad effect should be that cached data is discarded.
* demux_raw: set keyframe flagwm42019-09-191-0/+1
| | | | | | This is for uncompressed data, so every frame is a "keyframe". This is part of making this demuxer work with the demuxer layer caching and backward playback.
* Implement backwards playbackwm42019-09-194-23/+409
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | See manpage additions. This is a huge hack. You can bet there are shit tons of bugs. It's literally forcing square pegs into round holes. Hopefully, the manpage wall of text makes it clear enough that the whole shit can easily crash and burn. (Although it shouldn't literally crash. That would be a bug. It possibly _could_ start a fire by entering some sort of endless loop, not a literal one, just something where it tries to do work without making progress.) (Some obvious bugs I simply ignored for this initial version, but there's a number of potential bugs I can't even imagine. Normal playback should remain completely unaffected, though.) How this works is also described in the manpage. Basically, we demux in reverse, then we decode in reverse, then we render in reverse. The decoding part is the simplest: just reorder the decoder output. This weirdly integrates with the timeline/ordered chapter code, which also has special requirements on feeding the packets to the decoder in a non-straightforward way (it doesn't conflict, although a bugmessmass breaks correct slicing of segments, so EDL/ordered chapter playback is broken in backward direction). Backward demuxing is pretty involved. In theory, it could be much easier: simply iterating the usual demuxer output backward. But this just doesn't fit into our code, so there's a cthulhu nightmare of shit. To be specific, each stream (audio, video) is reversed separately. At least this means we can do backward playback within cached content (for example, you could play backwards in a live stream; on that note, it disables prefetching, which would lead to losing new live video, but this could be avoided). The fuckmess also meant that I didn't bother trying to support subtitles. Subtitles are a problem because they're "sparse" streams. They need to be "passively" demuxed: you don't try to read a subtitle packet, you demux audio and video, and then look whether there was a subtitle packet. This means to get subtitles for a time range, you need to know that you demuxed video and audio over this range, which becomes pretty messy when you demux audio and video backwards separately. Backward display is the most weird (and potentially buggy) part. To avoid that we need to touch a LOT of timing code, we negate all timestamps. The basic idea is that due to the navigation, all comparisons and subtractions of timestamps keep working, and you don't need to touch every single of them to "reverse" them. E.g.: bool before = pts_a < pts_b; would need to be: bool before = forward ? pts_a < pts_b : pts_a > pts_b; or: bool before = pts_a * dir < pts_b * dir; or if you, as it's implemented now, just do this after decoding: pts_a *= dir; pts_b *= dir; and then in the normal timing/renderer code: bool before = pts_a < pts_b; Consequently, we don't need many changes in the latter code. But some assumptions inhererently true for forward playback may have been broken anyway. What is mainly needed is fixing places where values are passed between positive and negative "domains". For example, seeking and timestamp user display always uses positive timestamps. The main mess is that it's not obvious which domain a given variable should or does use. Well, in my tests with a single file, it suddenly started to work when I did this. I'm honestly surprised that it did, and that I didn't have to change a single line in the timing code past decoder (just something minor to make external/cached text subtitles display). I committed it immediately while avoiding thinking about it. But there really likely are subtle problems of all sorts. As far as I'm aware, gstreamer also supports backward playback. When I looked at this years ago, I couldn't find a way to actually try this, and I didn't revisit it now. Back then I also read talk slides from the person who implemented it, and I'm not sure if and which ideas I might have taken from it. It's possible that the timestamp reversal is inspired by it, but I didn't check. (I think it claimed that it could avoid large changes by changing a sign?) VapourSynth has some sort of reverse function, which provides a backward view on a video. The function itself is trivial to implement, as VapourSynth aims to provide random access to video by frame numbers (so you just request decreasing frame numbers). From what I remember, it wasn't exactly fluid, but it worked. It's implemented by creating an index, and seeking to the target on demand, and a bunch of caching. mpv could use it, but it would either require using VapourSynth as demuxer and decoder for everything, or replacing the current file every time something is supposed to be played backwards. FFmpeg's libavfilter has reversal filters for audio and video. These require buffering the entire media data of the file, and don't really fit into mpv's architecture. It could be used by playing a libavfilter graph that also demuxes, but that's like VapourSynth but worse.
* demux: cleaner mutex usagewm42019-09-191-5/+6
| | | | | | | | | | | | | | The demuxer layer can start a thread to decouple the rest of the player from blocking I/O (such as network accesses). But this particular function does not support running with the thread enabled. The mutex use within it is only since thread_work() may temporarily unlock the mutex, and unlocking an unlocked mutex is not allowed. Most of the rest of the code still does proper locking, even if it's pointless and effectively single-threaded. To make this look slightly cleaner, extend the mutex around the rest of the code (like threaded code would have to do). This is mostly a cosmetic change.
* demux: add shitty start of stream detectionwm42019-09-191-8/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The demuxer cache benefits slightly from knowing where the current file or stream begins. For example, seeking "left most" when the start is cached would not trigger a low level seek (which would be followed by messy range joining when it notices that the newly demuxed packets overlap with an existing range). Unfortunately, since multimedia is so crazy (or actually FFmpeg in its quite imperfect attempt to be able to demux anything), it's hard to tell where a file starts. There is no feedback whether a specific seek went to the start of the file. Packets are not tagged with a flag indicating they were demuxed from the start position. There is no index available that could be used to cross-check this (even if the file contains a full and "perfect" index, like mp4). You could go by the timestamps, but who says streams start at 0? Streams can start somewhere at an extremely high timestamps (transport streams like to do that), or they could start at negative times (e.g. files with audio pre-padding will do that), and maybe some file formats simply allow negative timestamps and could start at any negative time. Even if the affected file formats don't allow it in theory, they may in practice. In addition, FFmpeg exports a start_time field, which may or may not be useful. (mpv's internal mkv demuxer also exports such a field, but doesn't bother to set it for efficiency and robustness reasons.) Anyway, this is all a huge load of crap, so I decided that if the user performs a seek command to time 0 or earlier, we consider the first packet demuxed from each stream to be at the start of the file. In addition, just trust the start_time field. This is the "shitty" part of this commit. One common case of negative timestamps is audio pre-padding. Demuxers normally behave sanely, and will treat 0 as the start of the file, and the first packets demuxed will have negative timestamps (since they contain data to discard), which doesn't break our assumptions in this commit. (Although, unfortunately, do break some other demuxer cache assumptions, and the first cached range will be shown as starting at a negative time.) Implementation-wise, this is quite simple. Just split the existing initial_state flag into two, since we want to deal with two separate aspects. In addition, this avoids the refresh seek on track switching when it happens right after a seek, instead of only after opening the demuxer.
* demux, command: export bof/eof flagswm42019-09-192-0/+3
| | | | | Export these flags with demuxer-cache-state. Useful for debugging, but any client API users could also make use of it.
* demux: remove logic duplication from packet read functionswm42019-09-191-68/+57
| | | | | | | | | | | | | | | | | There were 3 packet reading functions: the "old" demux_read_packet() that blocked (leftover from MPlayer times, but was still used until recently by some obscure code), the "new" demux_read_packet_async(), and the special demux_read_any_packet(), that is used by pseudo-demuxers like demux_edl. The first two could be used both in threaded and un-threaded mode. This made 5 cases in total. Some bits of logic was spread across all of them. Unify the logic. A recent commit made demux_read_packet() private, and the code for it in threaded mode disappears. The difference between threaded and un-threaded is minimized. It's possible that this commit causes random regression. Enjoy.
* sub: remove only user of demux_read_packet()wm42019-09-192-2/+1
| | | | | | | | | | | | | | | | | | | There are 3 packet reading functions in the demux API, which all function completely differently. One of them, demux_read_packet(), has only 1 caller, which is in dec_sub.c. Change this caller to use demux_read_packet_async() instead. Since it really wants to do a blocking call, setup some proper waiting. This uses mp_dispatch_queue, because even though it's overkill, it needs the least code. In practice, waiting actually never happens. This code is only called on code paths where everything is already read into memory (libavformat's subtitle demuxers simply behave this way). It's still a bit of a "coincidence", so implement it properly anyway. If suubtitle decoder init fails, we still need to unset the demuxer wakeup callback. Add a sub_destroy() call to the failure path. This also happens to fix a missed pthread_mutex_destroy() call (in practice this was a nop, or a memory leak on BSDs).
* demux: adjust reader_head on range joiningwm42019-09-191-0/+8
| | | | | | | | | | | | | | | | | | | I'm not sure about this, but it looks like a bug. If a stream didn't have packets, but the joined range does, the stream should obviously read the packets added by the joined range. Until now, due to reader_head being NULL, reading was only resumed if a _new_ packet was added by actual demuxing (in add_packet_locked()), which means the stream would suddenly skip ahead, past the original end of the joined range. Change it so that it will pick up the new range. Also, clear the skip_to_keyframe flag. Nothing useful can come from this flag being set; in the first place, the first packet of a range (that isn't the current range) should start with a keyframe. Some code probably enforced it (although it's fuzzy). Completely untested.
* demux: don't process obscure skipped packetswm42019-09-191-0/+5
| | | | | | | | | | When doing a seek to the end of the cache, ds->skip_to_keyframe can be set to true. Then some packets passed to add_packet_locked() may have to be skipped. In some aspects, the skipped packet was still treated as if it was going to be returned to the reader. It almost doesn't matter though: it only caused a redundant wakeup_ds() call, and could pass the packet to the stream recorder. Fix it anyway.
* demux: factor out a some packet queue codewm42019-09-191-11/+22
| | | | | Might be helpful for later. The "duplicated" ds->reader_head check above the function call is redundant, but leaving it also for later.
* demux: fix typos in commentswm42019-09-191-2/+2</