summaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAgeFilesLines
* demux_lavf: implement bad hack for backward playback of wavwm42019-09-192-15/+67
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit generally fixes backward playing in wav, at least in most PCM cases. libavformat's wav demuxer (and actually all other raw PCM based demuxers) have a specific behavior that breaks backward demuxing. The same thing also breaks persistent seek ranges in the demuxer cache, although that's less critical (it just means some cached data gets discarded). The backward demuxing issue is fatal, will log the message "Demuxer not cooperating.", and then typically stop doing anything. Unlike modern media formats, these formats don't organize media data in packets, but just wrap a monolithic byte stream that is described by a header. This is good enough for PCM, which uses fixed frames (a single sample for all audio channels), and for which it would be too expensive to have per frame headers. libavformat (and mpv) is heavily packet based, and using a single packet for each PCM frame causes too much overhead. So they typically "bundle" multiple frames into a single packet. This packet size is obviously arbitrary, and in libavformat's case hardcoded in its source code. The problem is that seeking doesn't respect this arbitrary packet boundary. Seeking is sample accurate. You can essentially seek inside a packet. The resulting packets will not be aligned with previously demuxed packets. This is normally OK. Backward seeking (and some other demuxer layer features) expect that demuxing an earlier demuxed file position eventually results in the same packets, regardless of the seeks that were done to get there. I like to call this "deterministic" demuxing. Backward demuxing in particular requires this to avoid overlaps, which would make it rather hard to get continuous output. Fix this issue by detecting wav and hopefully other raw audio formats with a heuristic (even PCM needs to be detected as heuristic). Then, if a seek is requested, align the seek timestamps on the guessed number of samples in the audio packets returned by the demuxer. The heuristic excludes files with multiple streams. (Except "attachment" video streams, which could be an ID3 tag. Yes, FFmpeg allows ID3 tags on WAV files.) Such files will inherently use the packet concept in some way. We don't know how the demuxer chooses the internal packet size, but we assume that it's fixed and aligned to PCM frame sizes. The frame size is most likely given by block_align (the native wav frame size, according to Microsoft). We possibly need to explicitly read and discard a packet if the seek is done without reading anything before that. We ignore any subsequent packet sizes; we need to avoid the very last packet, which likely has a different size. This hack should be rather benign. In the worst case, it will "round" the seek target a little, but the maximum rounding amount is bounded. Maybe we _could_ round up if SEEK_FORWARD is specified, but I didn't bother. An earlier commit fixed the same issue for mpv's demux_raw. An alternative, and probably much better solution would be clipping decoded data by timestamp. demux.c could allow the type of overlap the wav demuxer introduces, and instruct the decoder to clip the output against the last decoded timestamp. There's already an infrastructure for this (demux_packet.end field) used by EDL/ordered chapters. Although this sounds like a good solution, mpv unfortunately uses floats for timestamps. The rounding errors break sample accuracy. Even if you used integers, you'd need a timebase that is sample accurate (not always easy, since EDL can merge tracks with different sample rates).
* demux: add an explicit start state for backward demuxingwm42019-09-191-20/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Yay, more subtle state on top of this nightmarish, fragile state machine. But this is what happens when you subvert the laws of nature. This simple checks where playback should "resume" from when no packets were returned to the decoder yet after the seek that initiated backward playback. The main purpose is to process the first returned keyframe range in the same way like all other ranges. This ensures that things like preroll are included properly. Before this commit, it could for example have happened that the start of the first audio frame was slightly broken, because no preroll was included. Since the audio frame is reversed before sending it to the audio output, it would have added an audible discontinuity before the second frame was played; all subsequent frames would have been fine. (Although I didn't test and confirm this particular issue.) In future, this could be useful for certain other things. At least the condition for delaying the backstep seek becomes simpler and more explicit. Move the code that attempts to start demuxing up in dequeue_packet. Before, it was not called when the stream was in back_restarting state. This commit makes streams be in back_restarting state at initialization, so the demuxer would never have started reading. Likewise, we need to call back_demux_see_packets() right after seek in case the seek was within the cache. (We don't bother with checking whether it was a cached seek; nothing happens if it was a normal one.) There is nothing else that would process these cached packets explicitly, although coincidences could sporadically trigger it. The check for back_restart_next in find_backward_restart_pos() now decides whether to use this EOF special code. Since the backward playback start state also sets this variable, we don't need some of the complex checks in dequeue_packet() anymore either.
* manpage: deinterlacing with backwards playback probably workswm42019-09-191-3/+1
| | | | | | | | | | | | | | | | | | | | | | | As well as other filtering. I was writing this with the assumption that timestamps go backwards (which I first planned to do). But in fact, timestamps go forward, frame durations are positive, and adding a frame duration to a timestamp yields the correct result. The only strange thing is that timestamps are negative. Also, media of course goes backwards. In other possible implementation, filters would see normal forward playback, interrupted by seeks or discontinuities. It turns out the current implementation of providing a continuous backward media stream is probably better for filters. Even deinterlacing seems to work. libavcodec always outputs fields in as interleaved frames (i.e. fields are not reversed), and making up timestamps for the new frames (when doubling the framerate) works exactly like like in the forward case. Actually the previous paragraph was a lie, and libavcodec does not output fields as interleaved frames in rare cases. Sometimes AVFrame contains single fields. In this case you'd need to inverse the field dominance for deinterlacing filters to work correctly.
* manpage: backward encoding actually appears to workwm42019-09-191-1/+2
| | | | | | | | | | | | The way backward playback is implemented doesn't break basic assumptions about timestamps after the decoder, so I guess all the encoding mode needs to do is to adjust for the start offset, which it already does. Though I might be wrong and my test was possibly flawed. Stream recording on the other hand will fail immediately with --record-file, and --stream-record will probably yield unexpected results if any backstep seeks are done.
* demux: add a special case for backward demuxing of opuswm42019-09-192-2/+6
| | | | | | | | | | | Make --audio-backward-overlap default to 2 for Opus. I have no idea why this is needed. It seems to fix backward decoding though (going purely by listening). Normally, this should not be needed, since initial padding is completely contained within the first packet (normally, and in the case I tested). So the 2nd packet/frame should be fine, but for some unknown reason it works only with the 3rd.
* demux: use no overlapping packets for lossless audiowm42019-09-194-3/+14
| | | | | | Worthless optimization, but at least it justifies that the --audio-backward-overlap option has an "auto" choice. Tested with PCM and FLAC.
* manpage: document why Vorbis backward playback does not workwm42019-09-191-0/+7
| | | | | | | The only reasonable solution to this is probably to make discarding of preroll frames based on timestmaps, instead of frame/packet count. But then you get issues with video and its dumb timestamp reordering. So for now, fuck it.
* demux: remove some redundant pointer indirectionswm42019-09-191-13/+13
| | | | | | In all of these cases ds->in should be the same as the local variable in, and neither ds->in nor in ever change, i.e. a cosmetic simplification.
* demux: change backward-overlap to keyframe ranges instead of packetswm42019-09-192-39/+41
| | | | | | | | | This seems more useful in general. This change also happens to fix a miscounting of preroll packets when some of them were "rounded" away, and which could make it stuck. Also a simple intra-refresh encode with x264 (and muxed to mkv by it) seems to work now. I guess I misinterpreted earlier results.
* demux: fix typoswm42019-09-191-2/+2
|
* demux: redo backstep seek handling slightly againwm42019-09-191-2/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Backstepping still could get "stuck" if the demuxer didn't seek far back enough. This commit fixes getting stuck if playing backwards from the end, and audio has ended much earlier than the video. In commit "demux: fix initial backward demuxing state in some cases", I claimed that the backward seek semantics ("snapping" backward in normal seeking, unrelated to backward playing) would take care of this. Unfortunately, this is not always quite true. In theory, a seek to any position (that does not use SEEK_FORWARD, i.e. backward snapping) should return a packet for every stream. But I have a mkv sample, where audio ends much earlier than video. Its mkvmerge created index does not have entries for audio packets, so the video index is used. This index has its last entry somewhere close after the end of audio. So no audio packets will be returned. With a "too small" back_seek_size, the demuxer will retry a seek target that ends up in this place forever. (This does not happen if you use --index=recreate. It also doesn't happen with libavformat, which always prefers its own index, while mpv's internal mkv demuxer strictly prefers the index from the file if it can be read.) Fix this by adding the back_seek_size every time we fail to see enough packets. This way the seek step can add up until it works. To prevent that back_seek_pos just "runs away" towards negative infinity by subtracting back_seek_size every time we back step to undo forward reading (e.g. if --no-cache is used), readjust the back_seek_pos to the lowest known resume position. (If the cache is active, kf_seek_pts can be used, but to work in all situations, the code needs to grab the minimum PTS in the keyframe range.)
* demux: set SEEK_HR for backstep seeks, move a hr-seek detail to playloopwm42019-09-192-14/+9
| | | | | | Just rearranging shit. Setting SEEK_HR for backstep seeks actually doesn't have much meaning, but disables the weird audio snapping for "keyframe" seeks, and I don't know it's late.
* demux: rename a variablewm42019-09-191-8/+8
| | | | | It's "better". This is all what's left from an attempt to make the code slightly nicer.
* demux: remove minor code duplicationwm42019-09-191-16/+14
| | | | | | | | | | | | | | | | | | This code used to be simpler, but now it's enough that it should be factored into a single function. Both uses of the new function are annoyingly different. The first use is the special case when a decoder tries to read packets, but the demuxer doesn't see any (like mp4 files with sparse video packets, which actually turned out to be chapter thumbnail "tracks"). Then the other stream queues will overflow, and the stream with no packets is marked EOF to avoid stalling playback. The second case is when the demxuer returns global EOF. It would be more awkward to have the loop iterating the streams in the function, because then you'd need a weird parameter to control the behavior.
* demux: fix initial backward demuxing state in some caseswm42019-09-192-67/+95
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Just "mpv file.mkv --play-direction=backward" did not work, because backward demuxing from the very end was not implemented. This is another corner case, because the resume mechanism so far requires a packet "position" (dts or pos) as reference. Now "EOF" is another possible reference. Also, the backstep mechanism could cause streams to find different playback start positions, basically leading to random playback start (instead of what you specified with --start). This happens only if backstep seeks are involved (i.e. no cached data yet), but since this is usually the case at playback start, it always happened. It was racy too, because it depended on the order the decoders on other threads requested new data. The comment below "resume_earlier" has some more blabla. Some other details are changed. I'm giving up on the "from_cache" parameter, and don't try to detect the situation when the demuxer does not seek properly. Instead, always seek back, hopefully some more. Instead of trying to adjust the backstep seek target by a random value of 1.0 seconds. Instead, always rely on the random value provided by the user via --demuxer-backward-playback-step. If the demuxer should really get "stuck" and somehow miss the seek target badly, or the user sets the option value to 0, then the demuxer will not make any progress and just eat CPU. (Although due to backward seek semantics used for backstep seeks, even a very small seek step size will work. Just not 0.) It seems this also fixes backstepping correctly when the initial seek ended at the last keyframe range. (The explanation above was about the case when it ends at EOF. These two cases are different. In the former, you just need to step to the previous keyframe range, which was broken because it didn't always react correctly to reaching EOF. In the latter, you need to do a separate search for the last keyframe.)
* sd_lavc: implement --sub-pos for bitmap subtitleswm42019-09-192-0/+20
| | | | | | | | | | | | | Simple enough to do. May have mixed results. Typically, bitmap subtitles will have a tight bounding box around the rendered text. But if for example there is text on the top and bottom, it may be a single big bitmap with a large transparent area between top and bottom. In particular, DVD subtitles are really just a single screen-sized RLE-encoded bitmap, though libavcodec will crop off transparent areas. Like with sd_ass, you can't move subtitles _down_ if they are already in their origin position. This could probably be improved, but I don't want to deal with that right now.
* demux_lavf: also fix cache seeking with large codec delaywm42019-09-191-0/+2
| | | | | | | | | Fixes the same thing as the previous commit did with demux_mkv. I'm not sure if this is correct or a good idea (well, it works with my sample file). There are some shady things in this, but describing them would require too many expletives.
* demux, demux_mkv: fix seeking in cache with large codec delaywm42019-09-193-1/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | In this scenario, the demuxer will output timestamps offset by the codec delay (e.g. negative timestamps at the start; mkv simulates those), and the trimming in the decoder (often libavcodec, but ad_lavc.c in our case) will adjust the timestamps back (e.g. stream actually starts at 0). This offset needs to be taken into account when seeking. This worked in the uncached case. (demux_mkv.c is a bit tricky in that the index is already in the offset space, so it compensates even though the seek call does not reference codec_delay.) But in the cached case, seeks backwards did not seek enough, and forward they seeked too much. Fix this by adding the codec delay to the index search. We need to get "earlier" packets, so e.g. seeking to position 0 really gets the initial packets with negative timestamps. This also adjusts the seek range start. This is also pretty obvious: if the beginning of the file is cached, the seek range should start at 0, not a negative value. We compare 0-based timestamps to it later on. Not sure if this is the best approach. I also could have thought about/checked some corner cases harder. But fuck this shit. Not fixing duration (who cares) or end trimming, which would reduce the seek range and duration (who cares).
* demux_mkv: stop setting per-packet initial padding from codec delaywm42019-09-191-2/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | This is a bad approach, and should be handled by a codec parameter field (in mp_codec_params or AVCodecParameters). It's bad because it's overly complicated, and has potential to break demuxer cache assumptions: packets that were "intended" for seek resuming may suddenly appear in the middle of a stream, when you seek back and play a cached part again. (In general it was fine though, because seek range joining tends to remove the first audio packet of the next range when trying to find an overlap.) demux_mkv.c does not try to export its codec_delay field through the codec parameters mentioned above. In the only case I spotted this element, the codec itself (opus) set this field within libavcodec. And I think that's actually how it should be. On the other hand, a file could in theory set this field via mkv headers if the codec is too stupid to have such a field internally. But I don't really care until I see such a file. The end trimming is still sort of needed (though not sure if anything uses it, other than the opus/mkv test sample I was using). The decoder can't know whether something is the last packet, until it's too late. The codec_delay field is still needed to offset timestamps.
* ad_lavc: skip fully skipped frameswm42019-09-191-1/+5
| | | | | | | | | | | Fixes stupid messages with a opus/mkv test file that had an absurdly huge codec delay. This file fully skips several frames at the start. ad_lavc.c trimmed these frames to 0 samples and returned them. The next layer (f_decoder_wrapper.c) saw discontinuous PTS values, because the PTS values increased by a frame, but amounted to 0 audio samples. This was harmless, but logged PTS discontinuity errors.
* manpage: document that backward playback from the end does not workwm42019-09-191-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | Not specifying a --start or using --start=100% with --play-direction=backward usually does not work. The demuxer gets no packets and immediately enters EOF state, which then hangs because backward playback mode neither considers this mode, nor propagates the EOF. As far as demuxer implementations are concerned, this behavior is OK and even wanted. Seeking near the end with SEEK_FORWARD set is allowed not to return any packets (so a normal relative forward seek as done by the user would end playback). Seeking exactly to the end or past it without SEEK_FORWARD set is probably also sane. Another vaguely related issue is that a backward seek during playback start does not "establish" the demux position correctly: if stream A hits the next keyframe and seeks back, while stream B has not had a chance to read a packet yet, then stream B will never try to read from the old position. The effect is that stream B (and thus playback) will effectively miss the seek target. This is "random" because it depends on the order and number of packet read calls made by the decoders. Fixing this is probably hard, and requires extending the already complex state machine with more states, so turn the manpage into a TODO list for now.
* demux: don't adjust internal backward playback seeks by start timewm42019-09-191-3/+5
| | | | | | Only timestamps that enter or leave the demuxer API should be adjusted by ts_offset (which is usually the start time). queue_seek() is also used by backward demux seeks, which uses an internal timestamp.
* demux_raw: fix operation with demuxer cache and backward playbackwm42019-09-192-1/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Raw audio formats can be accessed sample-wise, and logically audio packets demuxed from it would contain only 1 sample. This is inefficient, so raw audio demuxers typically "bundle" multiple samples in one packet. The problem for the demuxer cache and backward playback is that they need properly aligned packets to make seeking "deterministic". The requirement is that if you read some packets, and then seek back, you eventually see the same packets again. demux_raw basically allowed to seek into the middle of a previously returned packet, which makes it impossible to make the transition seamless. (Unless you'd be aware of the packet data format and cut them to make it seamless, which is too complex for such a use case.) Solve this by always aligning seeks to packet boundaries. This reduces the seek accuracy to the arbitrarily chosen packet size. But you can use hr-seek to fix this. The gain from not making raw audio an awful special case pays in exchange for this "stupid" suggestion to use hr-seek. It appears this also fixes that it could and did seek into the middle of the frame (not sure if this code was ever tested - it goes back to removing the code duplication between the former demux_rawaudio.c and demux_rawvideo.c). If you really cared, you could introduce a seek flag that controls whether the seek is aligned or not. Then code which requires "deterministic" demuxing could set it. But this isn't really useful for us, and we'd always set the flag anyway, unless maybe the caching were forced disabled. libavformat's wav demuxer exhibits the same issue. We can't fix it (it would require the unpleasant experience of contributing to FFmpeg), so document this in otions.rst. In theory, this also affects seek range joining, but the only bad effect should be that cached data is discarded.
* demux_raw: set keyframe flagwm42019-09-191-0/+1
| | | | | | This is for uncompressed data, so every frame is a "keyframe". This is part of making this demuxer work with the demuxer layer caching and backward playback.
* sub: fix typo in commentwm42019-09-191-1/+1
|
* Implement backwards playbackwm42019-09-1922-43/+874
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | See manpage additions. This is a huge hack. You can bet there are shit tons of bugs. It's literally forcing square pegs into round holes. Hopefully, the manpage wall of text makes it clear enough that the whole shit can easily crash and burn. (Although it shouldn't literally crash. That would be a bug. It possibly _could_ start a fire by entering some sort of endless loop, not a literal one, just something where it tries to do work without making progress.) (Some obvious bugs I simply ignored for this initial version, but there's a number of potential bugs I can't even imagine. Normal playback should remain completely unaffected, though.) How this works is also described in the manpage. Basically, we demux in reverse, then we decode in reverse, then we render in reverse. The decoding part is the simplest: just reorder the decoder output. This weirdly integrates with the timeline/ordered chapter code, which also has special requirements on feeding the packets to the decoder in a non-straightforward way (it doesn't conflict, although a bugmessmass breaks correct slicing of segments, so EDL/ordered chapter playback is broken in backward direction). Backward demuxing is pretty involved. In theory, it could be much easier: simply iterating the usual demuxer output backward. But this just doesn't fit into our code, so there's a cthulhu nightmare of shit. To be specific, each stream (audio, video) is reversed separately. At least this means we can do backward playback within cached content (for example, you could play backwards in a live stream; on that note, it disables prefetching, which would lead to losing new live video, but this could be avoided). The fuckmess also meant that I didn't bother trying to support subtitles. Subtitles are a problem because they're "sparse" streams. They need to be "passively" demuxed: you don't try to read a subtitle packet, you demux audio and video, and then look whether there was a subtitle packet. This means to get subtitles for a time range, you need to know that you demuxed video and audio over this range, which becomes pretty messy when you demux audio and video backwards separately. Backward display is the most weird (and potentially buggy) part. To avoid that we need to touch a LOT of timing code, we negate all timestamps. The basic idea is that due to the navigation, all comparisons and subtractions of timestamps keep working, and you don't need to touch every single of them to "reverse" them. E.g.: bool before = pts_a < pts_b; would need to be: bool before = forward ? pts_a < pts_b : pts_a > pts_b; or: bool before = pts_a * dir < pts_b * dir; or if you, as it's implemented now, just do this after decoding: pts_a *= dir; pts_b *= dir; and then in the normal timing/renderer code: bool before = pts_a < pts_b; Consequently, we don't need many changes in the latter code. But some assumptions inhererently true for forward playback may have been broken anyway. What is mainly needed is fixing places where values are passed between positive and negative "domains". For example, seeking and timestamp user display always uses positive timestamps. The main mess is that it's not obvious which domain a given variable should or does use. Well, in my tests with a single file, it suddenly started to work when I did this. I'm honestly surprised that it did, and that I didn't have to change a single line in the timing code past decoder (just something minor to make external/cached text subtitles display). I committed it immediately while avoiding thinking about it. But there really likely are subtle problems of all sorts. As far as I'm aware, gstreamer also supports backward playback. When I looked at this years ago, I couldn't find a way to actually try this, and I didn't revisit it now. Back then I also read talk slides from the person who implemented it, and I'm not sure if and which ideas I might have taken from it. It's possible that the timestamp reversal is inspired by it, but I didn't check. (I think it claimed that it could avoid large changes by changing a sign?) VapourSynth has some sort of reverse function, which provides a backward view on a video. The function itself is trivial to implement, as VapourSynth aims to provide random access to video by frame numbers (so you just request decreasing frame numbers). From what I remember, it wasn't exactly fluid, but it worked. It's implemented by creating an index, and seeking to the target on demand, and a bunch of caching. mpv could use it, but it would either require using VapourSynth as demuxer and decoder for everything, or replacing the current file every time something is supposed to be played backwards. FFmpeg's libavfilter has reversal filters for audio and video. These require buffering the entire media data of the file, and don't really fit into mpv's architecture. It could be used by playing a libavfilter graph that also demuxes, but that's like VapourSynth but worse.
* demux: cleaner mutex usagewm42019-09-191-5/+6
| | | | | | | | | | | | | | The demuxer layer can start a thread to decouple the rest of the player from blocking I/O (such as network accesses). But this particular function does not support running with the thread enabled. The mutex use within it is only since thread_work() may temporarily unlock the mutex, and unlocking an unlocked mutex is not allowed. Most of the rest of the code still does proper locking, even if it's pointless and effectively single-threaded. To make this look slightly cleaner, extend the mutex around the rest of the code (like threaded code would have to do). This is mostly a cosmetic change.
* demux: add shitty start of stream detectionwm42019-09-191-8/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The demuxer cache benefits slightly from knowing where the current file or stream begins. For example, seeking "left most" when the start is cached would not trigger a low level seek (which would be followed by messy range joining when it notices that the newly demuxed packets overlap with an existing range). Unfortunately, since multimedia is so crazy (or actually FFmpeg in its quite imperfect attempt to be able to demux anything), it's hard to tell where a file starts. There is no feedback whether a specific seek went to the start of the file. Packets are not tagged with a flag indicating they were demuxed from the start position. There is no index available that could be used to cross-check this (even if the file contains a full and "perfect" index, like mp4). You could go by the timestamps, but who says streams start at 0? Streams can start somewhere at an extremely high timestamps (transport streams like to do that), or they could start at negative times (e.g. files with audio pre-padding will do that), and maybe some file formats simply allow negative timestamps and could start at any negative time. Even if the affected file formats don't allow it in theory, they may in practice. In addition, FFmpeg exports a start_time field, which may or may not be useful. (mpv's internal mkv demuxer also exports such a field, but doesn't bother to set it for efficiency and robustness reasons.) Anyway, this is all a huge load of crap, so I decided that if the user performs a seek command to time 0 or earlier, we consider the first packet demuxed from each stream to be at the start of the file. In ad