summaryrefslogtreecommitdiffstats
path: root/demux
Commit message (Collapse)AuthorAgeFilesLines
* demux: update some commentswm42019-09-191-15/+28
| | | | | | | Mostly about the packet queue and the subtitle handling of it. (This mess sure sounds like a good argument to give up the separate stream queues, and using a single packet queue per cached range.)
* demux: shorten some redundant outputwm42019-09-191-3/+1
| | | | This message would always show "correct_dts=0 correct_pos=0".
* demux: demux multiple audio frames in backward playbackwm42019-09-191-38/+65
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Until now, this usually passed a single audio frame to the decoder, and then did a backstep operation (cache seek + frame search) again. This is probably not very efficient, especially considering it has to search the packet queue from the "start" every time again. Also, with most audio codecs, an additional "preroll" frame was passed first. In these cases, the preroll frame would make up 50% of audio decoding time. Also not very efficient. Attempt to fix this by returning multiple frames at once. This reduces the number of backstep operations and the ratio the preoll frames. In theory, this should help efficiency. I didn't test it though, why would I do this? It's just a pain. Set it to unscientific 10 frames. (Actually, these are 10 keyframes, so it's much more for codecs like TrueHD. But I don't care about TrueHD.) This commit changes some other implementation details. Since we can return more than 1 non-preroll keyframe to the decoder, some new state is needed to remember how much. The resume packet search is adjusted to find N ("total") keyframe packets in general, not just preroll frames. I'm removing the special case for 1 preroll packet; audio used this, but doesn't anymore, and it's premature optimization anyway. Expose the new mechanism with 2 new options. They're almost completely pointless, since nobody will try them, and if they do, they won't understand what these options truly do. And if they actually do, they most likely would be capable of editing the source code, and we could just hardcode the parameters. Just so you know that I know that the added options are pointless. The following two things are truly unrelated to this commit, and more like general refactoring, but fortunately nobody can stop me. Don't set back_seek_pos in dequeue_packet() anymore. This was sort of pointless, since it was set in find_backward_restart_pos() anyway (using some of the same packets). The latter function tries to restrict this to the first keyframe range though, which is an optimization that in theory might break with broken files (duh), but in these cases a lot of other things would be broken anyway. Don't set back_restart_* in dequeue_packet(). I think this is an artifact of the old restart code (cf. ad9e473c555). It can be done directly in find_backward_restart_pos() now. Although this adds another shitty packet search loop, I prefer this, because clearer what's actually happening.
* demux: remove further calls to packet size estimation functionwm42019-09-191-2/+5
| | | | May as well be part of the previous commit.
* demux: don't loop over all packets to find forward buffered size on seekwm42019-09-192-46/+50
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The size of all forward buffered packets is used to control maximum buffering. Until now, this size was incrementally adjusted, but had to be recomputed on seeks within the cache. Doing this was actually pretty expensive. It iterates over a linked list of separate memory allocations (which are probably spread all over the heap due to the allocation behavior), and the demux_packet_estimate_total_size() call touches a lot of further memory locations. I guess this affects the cache rather negatively. In an unscientific test, the recompute_buffers() function (which contained this loop) was responsible for roughly half of the time seeking took. Replace this with a way that computes the buffered size between 2 packets in constant times. The demux_packet.cum_pos field contains the summed sizes of all previous packets, so subtracting cum_pos between two packets yields the size of all packets in between. We can do this because we never remove packets from the middle of the queue. We only add packets to the end, or remove packets at the beginning. The tail_cum_pos field is needed because we don't store the end position of a packet, so the last packet's position would be unknown. We could recompute the "estimated" packet size, or store the estimated size in the packet struct, but I just didn't like this. This also removes the cached fw_bytes fields. It's slightly nicer to just recompute them when needed. Maintaining them incrementally was annoying. total_size stays though, since recomputing it isn't that cheap (would need to loop over all ranges every time). I'm always using uint64_t for sizes. This is certainly needed (a stream could easily burn through more than 4GB of data, even if much less of that is cached). The actual cached amount should always fit into size_t, so it's casted to size_t for printfs (yes, I hate the way you specify stdint.h types in printfs, the less I have to use that crap, the better).
* demux: remove tracking of number of forward buffered packetswm42019-09-191-14/+10
| | | | | | | | | | | | | | In ancient times, the number of packets was used to limit excessive read-ahead. This was completely replaced by tracking the size in bytes. The number of packets was used in debugging output only. In one case (packet got demuxed and is added to a queue), only log whether there were packets on this stream before. (Unknown whether it's useful.) In another case (queue overflow), actually count the number of packets. It's vaguely useful, and the message with the number of packets is shown only once after a seek reset, so it doesn't matter whether it's slow.
* demux: fix backward demuxing freeze if first packet is not a keyframewm42019-09-191-1/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Some files don't start with keyframe packets. Normally, this is not sane, but the sample file which triggered this was a cut TV capture transport stream. And this shouldn't happen anyway. Introduce a further heuristic: if the last seek target was before the start of the cached data, and the start of the cache is marked as BOF (beginning of file), then we won't find anything better. This is possibly a bit shaky, because both seek_start and back_seek_pos weren't made for this purpose. But I can't come up with situations where this would actually break. (Leave this to shitty broken files I hit later.) I also considered finding the first packet in the cache that is marked as keyframe, i.e. the first actual seek target, and comparing it to "first", but I didn't like it much. Well whatever. It's a bit silly that this caused a hard freeze (and similar issues still will). The problem is that the demuxer holds the lock and has no reason to release it. And in general, there's a single lock for the entire demuxer cache. Finer grained locking would probably not make much sense. In theory status of available data and maybe certain commands to the demuxer could be moved to separate locks, but it would raise complexity, and you'd probably still need to get the central lock in some cases, which would deadlock you anyway. It would still be nice if some minor corner case in the wonderfully terrible and complex backward demuxer state machine couldn't lock up the player. As a hack, unlock and then immediately lock again. Depending on the OS mutex implementation, this may give other waiters a chance to grab the lock. This is not a guarantee (some OSes may for example not wake up other waiters until the next time slice or something), but works well on Linux.
* demux: simplify and improve performance of backward playback steppingwm42019-09-191-33/+6
| | | | | | | | | | | | | | | | | The step_backwards function set reader_head to the start of the current cache range. This was completely unnecessary and made it _much_ slower. Remove the code that adjusts reader_head. Merge the rest of the code into the only caller and remove the function. The comment on the removed code was quite right. It was "inefficient". Removing it delegates going to an early position to the normal seek code, triggered by find_backward_restart_pos() incremental back seek logic. I suppose especially audio benefits from this, because this happens for every single audio packet (except maybe freaky bullshit like TrueHD, which has "keyframes"). The blabla about performance in the removed comments is still true, but now applies to the seek code itself only.
* demux: fix backward playback at EOF with full demuxer cachewm42019-09-191-3/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | Fixes "mpv file.mkv --cache --demuxer-cache-wait --play-dir=backward", and other situations where the demuxer cache contains the entire file, and playback is to start from the end. It also can be triggered when starting playback normally with --cache, and once everything is in the cache, enabling backward playback and seeking past EOF. In all cases, the cache seek will set reader_head=NULL (because you seeked to/past EOF). Then the code (the one modified by this commit) sees that ds->queue->is_bof==true, and thinks we've reached BOF (beginning of file) while searching for a useful packet, i.e. we found nothing and playback really can only end. Obviously this is nonsense, we've found only nothing if we actually searched from the beginning, not some "random" reader_head (== first) value that does not include the entire cache. That means the condition should trigger only if the start of the search (first variable) points to the beginning of the cache (ds->queue->head). Not taking this if means we'll seek to an earlier position and retry. Also, a seek before the beginning of the cache will always end up with reader_head==ds->queue->head, i.e. we'll terminate properly. That comment was quite right.
* demux: more backwards playback preroll packets for vorbis and mp3wm42019-09-191-1/+3
| | | | | | | | | | | | | | | | | | | | Together with the previous commit, this seems to make backward playback work in files with vorbis and mp3 audio codecs. For Vorbis (with libavcodec's decoder, didn't test libvorbis), the first packet was just always completely discarded. This happened even though we tell libavcodec that we do discarding of padding manually. It simply happened inside the codec, not libavcodec's general initial padding handling. In addition, the first output decoded frame seems to contain partial data. (Unlike the opus decoder, it doesn't report any padding at all.) The Opus decoder (again libavcodec only tested) reports an initial padding, but it appears to be too small, and it sounds right only with 2 packets discarded. So its status doesn't change. I'm not sure why I need 2 frames for mp3, but with that value I had success on the samples I tested.
* demux_mkv: don't set keyframe flag for timestamp-less audio frameswm42019-09-191-2/+3
| | | | | | | | | | | | | | | | | | | | | Matroska has this weird concept of "lacing", which are really sub-blocks packed into a larger actual block. They are demuxed as individual packets, because that's what the decoder needs. Basically they're a Matroska hack to reduce per-packet muxing overhead. One problem is that these sub-blocks don't have timestamps. The timestamps need to be created from the "default duration". If this default duration isn't in the file header (or if we drop it when it has a known broken value), the resulting packets won't have a timestamp. This is an usual and weird situation, that may confuse the demuxer layer (and backward playback in particular). Fix this by not setting the keyframe flag for these. This affects only audio packets. Subtitle lacing is explicitly not supported (in theory would at least lack a way to specify durations), and video won't usually use it for real video codecs (not every frame is a "keyframe", timestamp reordering).
* demux: move timestamp helper macros to common.hwm42019-09-191-12/+4
| | | | These are probably generally useful.
* demux, f_decoder_wrapper: fix coverart in backward modewm42019-09-191-11/+11
| | | | | | | | | | | | Shitty ancient hack that wastes my time all the time. demux.c: always return the coverart packet as soon as possible, and don't let the backward demux state machine possibly stop it. f_decoder_wrapper.c: mess with some shit until it somehow starts to work. I think the old code tried to let it cleverly fall through so the packet was processed "normally"; just make it run the "usual" code instead.
* demux_lavf: implement bad hack for backward playback of wavwm42019-09-191-4/+64
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit generally fixes backward playing in wav, at least in most PCM cases. libavformat's wav demuxer (and actually all other raw PCM based demuxers) have a specific behavior that breaks backward demuxing. The same thing also breaks persistent seek ranges in the demuxer cache, although that's less critical (it just means some cached data gets discarded). The backward demuxing issue is fatal, will log the message "Demuxer not cooperating.", and then typically stop doing anything. Unlike modern media formats, these formats don't organize media data in packets, but just wrap a monolithic byte stream that is described by a header. This is good enough for PCM, which uses fixed frames (a single sample for all audio channels), and for which it would be too expensive to have per frame headers. libavformat (and mpv) is heavily packet based, and using a single packet for each PCM frame causes too much overhead. So they typically "bundle" multiple frames into a single packet. This packet size is obviously arbitrary, and in libavformat's case hardcoded in its source code. The problem is that seeking doesn't respect this arbitrary packet boundary. Seeking is sample accurate. You can essentially seek inside a packet. The resulting packets will not be aligned with previously demuxed packets. This is normally OK. Backward seeking (and some other demuxer layer features) expect that demuxing an earlier demuxed file position eventually results in the same packets, regardless of the seeks that were done to get there. I like to call this "deterministic" demuxing. Backward demuxing in particular requires this to avoid overlaps, which would make it rather hard to get continuous output. Fix this issue by detecting wav and hopefully other raw audio formats with a heuristic (even PCM needs to be detected as heuristic). Then, if a seek is requested, align the seek timestamps on the guessed number of samples in the audio packets returned by the demuxer. The heuristic excludes files with multiple streams. (Except "attachment" video streams, which could be an ID3 tag. Yes, FFmpeg allows ID3 tags on WAV files.) Such files will inherently use the packet concept in some way. We don't know how the demuxer chooses the internal packet size, but we assume that it's fixed and aligned to PCM frame sizes. The frame size is most likely given by block_align (the native wav frame size, according to Microsoft). We possibly need to explicitly read and discard a packet if the seek is done without reading anything before that. We ignore any subsequent packet sizes; we need to avoid the very last packet, which likely has a different size. This hack should be rather benign. In the worst case, it will "round" the seek target a little, but the maximum rounding amount is bounded. Maybe we _could_ round up if SEEK_FORWARD is specified, but I didn't bother. An earlier commit fixed the same issue for mpv's demux_raw. An alternative, and probably much better solution would be clipping decoded data by timestamp. demux.c could allow the type of overlap the wav demuxer introduces, and instruct the decoder to clip the output against the last decoded timestamp. There's already an infrastructure for this (demux_packet.end field) used by EDL/ordered chapters. Although this sounds like a good solution, mpv unfortunately uses floats for timestamps. The rounding errors break sample accuracy. Even if you used integers, you'd need a timebase that is sample accurate (not always easy, since EDL can merge tracks with different sample rates).
* demux: add an explicit start state for backward demuxingwm42019-09-191-20/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Yay, more subtle state on top of this nightmarish, fragile state machine. But this is what happens when you subvert the laws of nature. This simple checks where playback should "resume" from when no packets were returned to the decoder yet after the seek that initiated backward playback. The main purpose is to process the first returned keyframe range in the same way like all other ranges. This ensures that things like preroll are included properly. Before this commit, it could for example have happened that the start of the first audio frame was slightly broken, because no preroll was included. Since the audio frame is reversed before sending it to the audio output, it would have added an audible discontinuity before the second frame was played; all subsequent frames would have been fine. (Although I didn't test and confirm this particular issue.) In future, this could be useful for certain other things. At least the condition for delaying the backstep seek becomes simpler and more explicit. Move the code that attempts to start demuxing up in dequeue_packet. Before, it was not called when the stream was in back_restarting state. This commit makes streams be in back_restarting state at initialization, so the demuxer would never have started reading. Likewise, we need to call back_demux_see_packets() right after seek in case the seek was within the cache. (We don't bother with checking whether it was a cached seek; nothing happens if it was a normal one.) There is nothing else that would process these cached packets explicitly, although coincidences could sporadically trigger it. The check for back_restart_next in find_backward_restart_pos() now decides whether to use this EOF special code. Since the backward playback start state also sets this variable, we don't need some of the complex checks in dequeue_packet() anymore either.
* demux: add a special case for backward demuxing of opuswm42019-09-191-1/+4
| | | | | | | | | | | Make --audio-backward-overlap default to 2 for Opus. I have no idea why this is needed. It seems to fix backward decoding though (going purely by listening). Normally, this should not be needed, since initial padding is completely contained within the first packet (normally, and in the case I tested). So the 2nd packet/frame should be fine, but for some unknown reason it works only with the 3rd.
* demux: use no overlapping packets for lossless audiowm42019-09-191-2/+3
| | | | | | Worthless optimization, but at least it justifies that the --audio-backward-overlap option has an "auto" choice. Tested with PCM and FLAC.
* demux: remove some redundant pointer indirectionswm42019-09-191-13/+13
| | | | | | In all of these cases ds->in should be the same as the local variable in, and neither ds->in nor in ever change, i.e. a cosmetic simplification.
* demux: change backward-overlap to keyframe ranges instead of packetswm42019-09-191-34/+36
| | | | | | | | | This seems more useful in general. This change also happens to fix a miscounting of preroll packets when some of them were "rounded" away, and which could make it stuck. Also a simple intra-refresh encode with x264 (and muxed to mkv by it) seems to work now. I guess I misinterpreted earlier results.
* demux: fix typoswm42019-09-191-2/+2
|
* demux: redo backstep seek handling slightly againwm42019-09-191-2/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Backstepping still could get "stuck" if the demuxer didn't seek far back enough. This commit fixes getting stuck if playing backwards from the end, and audio has ended much earlier than the video. In commit "demux: fix initial backward demuxing state in some cases", I claimed that the backward seek semantics ("snapping" backward in normal seeking, unrelated to backward playing) would take care of this. Unfortunately, this is not always quite true. In theory, a seek to any position (that does not use SEEK_FORWARD, i.e. backward snapping) should return a packet for every stream. But I have a mkv sample, where audio ends much earlier than video. Its mkvmerge created index does not have entries for audio packets, so the video index is used. This index has its last entry somewhere close after the end of audio. So no audio packets will be returned. With a "too small" back_seek_size, the demuxer will retry a seek target that ends up in this place forever. (This does not happen if you use --index=recreate. It also doesn't happen with libavformat, which always prefers its own index, while mpv's internal mkv demuxer strictly prefers the index from the file if it can be read.) Fix this by adding the back_seek_size every time we fail to see enough packets. This way the seek step can add up until it works. To prevent that back_seek_pos just "runs away" towards negative infinity by subtracting back_seek_size every time we back step to undo forward reading (e.g. if --no-cache is used), readjust the back_seek_pos to the lowest known resume position. (If the cache is active, kf_seek_pts can be used, but to work in all situations, the code needs to grab the minimum PTS in the keyframe range.)
* demux: set SEEK_HR for backstep seeks, move a hr-seek detail to playloopwm42019-09-191-10/+1
| | | | | | Just rearranging shit. Setting SEEK_HR for backstep seeks actually doesn't have much meaning, but disables the weird audio snapping for "keyframe" seeks, and I don't know it's late.
* demux: rename a variablewm42019-09-191-8/+8
| | | | | It's "better". This is all what's left from an attempt to make the code slightly nicer.
* demux: remove minor code duplicationwm42019-09-191-16/+14
| | | | | | | | | | | | | | | | | | This code used to be simpler, but now it's enough that it should be factored into a single function. Both uses of the new function are annoyingly different. The first use is the special case when a decoder tries to read packets, but the demuxer doesn't see any (like mp4 files with sparse video packets, which actually turned out to be chapter thumbnail "tracks"). Then the other stream queues will overflow, and the stream with no packets is marked EOF to avoid stalling playback. The second case is when the demxuer returns global EOF. It would be more awkward to have the loop iterating the streams in the function, because then you'd need a weird parameter to control the behavior.
* demux: fix initial backward demuxing state in some caseswm42019-09-191-59/+95
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Just "mpv file.mkv --play-direction=backward" did not work, because backward demuxing from the very end was not implemented. This is another corner case, because the resume mechanism so far requires a packet "position" (dts or pos) as reference. Now "EOF" is another possible reference. Also, the backstep mechanism could cause streams to find different playback start positions, basically leading to random playback start (instead of what you specified with --start). This happens only if backstep seeks are involved (i.e. no cached data yet), but since this is usually the case at playback start, it always happened. It was racy too, because it depended on the order the decoders on other threads requested new data. The comment below "resume_earlier" has some more blabla. Some other details are changed. I'm giving up on the "from_cache" parameter, and don't try to detect the situation when the demuxer does not seek properly. Instead, always seek back, hopefully some more. Instead of trying to adjust the backstep seek target by a random value of 1.0 seconds. Instead, always rely on the random value provided by the user via --demuxer-backward-playback-step. If the demuxer should really get "stuck" and somehow miss the seek target badly, or the user sets the option value to 0, then the demuxer will not make any progress and just eat CPU. (Although due to backward seek semantics used for backstep seeks, even a very small seek step size will work. Just not 0.) It seems this also fixes backstepping correctly when the initial seek ended at the last keyframe range. (The explanation above was about the case when it ends at EOF. These two cases are different. In the former, you just need to step to the previous keyframe range, which was broken because it didn't always react correctly to reaching EOF. In the latter, you need to do a separate search for the last keyframe.)
* demux_lavf: also fix cache seeking with large codec delaywm42019-09-191-0/+2
| | | | | | | | | Fixes the same thing as the previous commit did with demux_mkv. I'm not sure if this is correct or a good idea (well, it works with my sample file). There are some shady things in this, but describing them would require too many expletives.
* demux, demux_mkv: fix seeking in cache with large codec delaywm42019-09-193-1/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | In this scenario, the demuxer will output timestamps offset by the codec delay (e.g. negative timestamps at the start; mkv simulates those), and the trimming in the decoder (often libavcodec, but ad_lavc.c in our case) will adjust the timestamps back (e.g. stream actually starts at 0). This offset needs to be taken into account when seeking. This worked in the uncached case. (demux_mkv.c is a bit tricky in that the index is already in the offset space, so it compensates even though the seek call does not reference codec_delay.) But in the cached case, seeks backwards did not seek enough, and forward they seeked too much. Fix this by adding the codec delay to the index search. We need to get "earlier" packets, so e.g. seeking to position 0 really gets the initial packets with negative timestamps. This also adjusts the seek range start. This is also pretty obvious: if the beginning of the file is cached, the seek range should start at 0, not a negative value. We compare 0-based timestamps to it later on. Not sure if this is the best approach. I also could have thought about/checked some corner cases harder. But fuck this shit. Not fixing duration (who cares) or end trimming, which would reduce the seek range and duration (who cares).
* demux_mkv: stop setting per-packet initial padding from codec delaywm42019-09-191-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | This is a bad approach, and should be handled by a codec parameter field (in mp_codec_params or AVCodecParameters). It's bad because it's overly complicated, and has potential to break demuxer cache assumptions: packets that were "intended" for seek resuming may suddenly appear in the middle of a stream, when you seek back and play a cached part again. (In general it was fine though, because seek range joining tends to remove the first audio packet of the next range when trying to find an overlap.) demux_mkv.c does not try to export its codec_delay field through the codec parameters mentioned above. In the only case I spotted this element, the codec itself (opus) set this field within libavcodec. And I think that's actually how it should be. On the other hand, a file could in theory set this field via mkv headers if the codec is too stupid to have such a field internally. But I don't really care until I see such a file. The end trimming is still sort of needed (though not sure if anything uses it, other than the opus/mkv test sample I was using). The decoder can't know whether something is the last packet, until it's too late. The codec_delay field is still needed to offset timestamps.
* demux: don't adjust internal backward playback seeks by start timewm42019-09-191-3/+5
| | | | | | Only timestamps that enter or leave the demuxer API should be adjusted by ts_offset (which is usually the start time). queue_seek() is also used by backward demux seeks, which uses an internal timestamp.
* demux_raw: fix operation with demuxer cache and backward playbackwm42019-09-191-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Raw audio formats can be accessed sample-wise, and logically audio packets demuxed from it would contain only 1 sample. This is inefficient, so raw audio demuxers typically "bundle" multiple samples in one packet. The problem for the demuxer cache and backward playback is that they need properly aligned packets to make seeking "deterministic". The requirement is that if you read some packets, and then seek back, you eventually see the same packets again. demux_raw basically allowed to seek into the middle of a previously returned packet, which makes it impossible to make the transition seamless. (Unless you'd be aware of the packet data format and cut them to make it seamless, which is too complex for such a use case.) Solve this by always aligning seeks to packet boundaries. This reduces the seek accuracy to the arbitrarily chosen packet size. But you can use hr-seek to fix this. The gain from not making raw audio an awful special case pays in exchange for this "stupid" suggestion to use hr-seek. It appears this also fixes that it could and did seek into the middle of the frame (not sure if this code was ever tested - it goes back to removing the code duplication between the former demux_rawaudio.c and demux_rawvideo.c). If you really cared, you could introduce a seek flag that controls whether the seek is aligned or not. Then code which requires "deterministic" demuxing could set it. But this isn't really useful for us, and we'd always set the flag anyway, unless maybe the caching were forced disabled. libavformat's wav demuxer exhibits the same issue. We can't fix it (it would require the unpleasant experience of contributing to FFmpeg)