path: root/DOCS/tech/nut.txt

==================================
NUT Open Container Format 20061104
==================================



Intro:
======

NUT is a free multimedia container format for storage of audio, video,
subtitles and related user defined streams, it provides exact timestamps for
synchronization and seeking, is simple, has low overhead and can recover
in case of errors in the stream.

Other common multimedia container formats are AVI, Ogg, Matroska, MP4, MOV
ASF, MPEG-PS, MPEG-TS.


Features / goals:
    (supported by the format, not necessarily by a specific implementation)

Simplicity
    Use the same encoding for nearly all fields.
    Simple decoding, so slow CPUs (and embedded systems) can handle it.

Extensibility
    No limit for the possible values of all fields (using universal vlc).
    Allow adding of new headers in the future.
    Allow adding more fields at the end of headers.

Compactness
    ~0.2% overhead for normal bitrates.
    The index is <100kb per hour.
    A typical file header is about 100 bytes (audio + video headers together).
    A packet header is about ~1-5 bytes.

Error resistance
    Seeking / playback is possible without an index.
    Headers & index can be repeated.
    Damaged files can be played back with minimal data loss and fast
    resynchronization times.

The specification is frozen. All files following the specification will be
compatible unless the specification is unfrozen.


Definitions:
============

MUST    The specific part must be done to conform to this standard.
SHOULD  It is recommended to be done that way, but not strictly required.

keyframe
    A keyframe is a frame from which you can start decoding, a more
    exact definition is below
    The nth frame is a keyframe if and only if frames n, n+1, ... in
    presentation order (that are all frames with a pts >= frame[n].pts) can
    be decoded successfully without reference to frames prior n in storage
    order (that are all frames with a dts < frame[n].dts).
    If no such frames exist (for example due to using overlapped transforms
    like the MDCT in an audio codec), then the definition shall be extended
    by dropping n out of the set of frames which must be decodable, if this
    is still insufficient then n+1 shall be dropped, and so on until there is
    a keyframe.
    Every frame which is marked as a keyframe MUST be a keyframe according to
    the definition above, a muxer MUST mark every frame it knows is a keyframe
    as such, a muxer SHOULD NOT analyze future frames to determine the
    keyframe status of the current frame but instead just set the frame as
    non-keyframe.
    (FIXME maybe move somewhere else?)
pts
    Presentation time of the first frame/sample that is completed by decoding
    the coded frame.
dts
    The time when a frame is input into a synchronous 1-in-1-out decoder.


Syntax:
=======

Since NUT heavily uses variable length fields, the simplest way to describe it
is using a pseudocode approach.



Conventions:
============

The data types have a name, used in the bitstream syntax description, a short
text description and a pseudocode (functional) definition, optional notes may
follow:

name    (text description)
    functional definition
    [Optional notes]

The bitstream syntax elements have a tagname and a functional definition, they
are presented in a bottom-up approach, again optional notes may follow and
are reproduced in the tag description:

name:    (optional note)
    functional definition
    [Optional notes]

The in-depth tag description follows the bitstream syntax.
The functional definition has a C-like syntax.



Type definitions:
=================

f(n)    (n fixed bits in big-endian order)
u(n)    (unsigned number encoded in n bits in MSB-first order)

v   (variable length value, unsigned)
    value=0
    do{
        more_data                       u(1)
        data                            u(7)
        value= 128*value + data
    }while(more_data)

s   (variable length value, signed)
    temp                                v
    temp++
    if(temp&1) value= -(temp>>1)
    else       value=  (temp>>1)

b   (binary data or string, to be use in vb, see below)
    for(i=0; i<length; i++){
        data[i]                         u(8)
    }
    [Note: strings MUST be encoded in UTF-8]
    [Note: the character NUL (U+0000) is not legal within
    or at the end of a string.]

vb  (variable length binary data or string)
    length                              v
    value                               b

t (v coded universal timestamp)
    tmp                                 v
    id= tmp % time_base_count
    value= (tmp / time_base_count) * time_base[id]


Bitstream syntax:
=================

file:
    file_id_string
    while(!eof){
        if(next_byte == 'N'){
            packet_header
            switch(startcode){
                case      main_startcode:  main_header; break;
                case    stream_startcode:stream_header; break;
                case      info_startcode:  info_packet; break;
                case     index_startcode:        index; break;
                case syncpoint_startcode:    syncpoint; break;
            }
            packet_footer
        }else
            frame
    }

The structure of an undamaged file should look like the following, but
demuxers should be flexible and be able to deal with damaged headers so the
above is a better loop in practice (not to mention it is simpler).
Note: Demuxers MUST be able to deal with new and unknown headers.

file:
    file_id_string
    while(!eof){
        packet_header, main_header, packet_footer
        reserved_headers
        for(i=0; i<stream_count; i++){
            packet_header, stream_header, packet_footer
            reserved_headers
        }
        while(next_code == info_startcode){
            packet_header, info_packet, packet_footer
            reserved_headers
        }
        if(next_code == index_startcode){
            packet_header, index_packet, packet_footer
        }
        if (!eof) while(next_code != main_startcode){
            if(next_code == syncpoint_startcode){
                packet_header, syncpoint, packet_footer
            }
            frame
            reserved_headers
        }
    }


Common elements:
----------------

reserved_bytes:
    for(i=0; i<forward_ptr - length_of_non_reserved; i++)
        reserved                        u(8)
    [A demuxer MUST ignore any reserved bytes.
    A muxer MUST NOT write any reserved bytes, as this would make it
    impossible to add new fields at the end of packets in the future
    in a compatible way.]

packet_header
    startcode                           f(64)
    forward_ptr                         v
    if(forward_ptr > 4096)
        header_checksum                 u(32)

packet_footer
    checksum                            u(32)

reserved_headers
    while(next_byte == 'N' && next_code !=      main_startcode
                           && next_code !=    stream_startcode
                           && next_code !=      info_startcode
                           && next_code !=     index_startcode
                           && next_code != syncpoint_startcode){
        packet_header
        reserved_bytes
        packet_footer
    }

        Headers:

main_header:
    version                             v
    stream_count                        v
    max_distance                        v
    time_base_count                     v
    for(i=0; i<time_base_count; i++)
        time_base_num                   v
        time_base_denom                 v
        time_base[i]= time_base_num/time_base_denom
    tmp_pts=0
    tmp_mul=1
    tmp_stream=0
    for(i=0; i<256; ){
        tmp_flag                        v
        tmp_fields                      v
        if(tmp_fields>0) tmp_pts        s
        if(tmp_fields>1) tmp_mul        v
        if(tmp_fields>2) tmp_stream     v
        if(tmp_fields>3) tmp_size       v
        else tmp_size=0
        if(tmp_fields>4) tmp_res        v
        else tmp_res=0
        if(tmp_fields>5) count          v
        else count= tmp_mul - tmp_size
        for(j=6; j<tmp_fields; j++){
            tmp_reserved[i]             v
        }
        for(j=0; j<count && i<256; j++, i++){
            if (i == 'N') {
                flags[i]= FLAG_INVALID;
                j--;
                continue;
            }
            flags[i]= tmp_flag;
            stream_id[i]= tmp_stream;
            data_size_mul[i]= tmp_mul;
            data_size_lsb[i]= tmp_size + j;
            pts_delta[i]= tmp_pts;
            reserved_count[i]= tmp_res;
        }
    }
    reserved_bytes

stream_header:
    stream_id                           v
    stream_class                        v
    fourcc                              vb
    time_base_id                        v
    msb_pts_shift                       v
    max_pts_distance                    v
    decode_delay                        v
    stream_flags                        v
    codec_specific_data                 vb
    if(stream_class == video){
        width                           v
        height                          v
        sample_width                    v
        sample_height                   v
        colorspace_type                 v
    }else if(stream_class == audio){
        samplerate_num                  v
        samplerate_denom                v
        channel_count                   v
    }
    reserved_bytes

        Basic Packets:

frame:
    frame_code                          f(8)
    frame_flags= flags[frame_code]
    frame_res= reserved_count[frame_code]
    if(frame_flags&FLAG_CODED){
        coded_flags                     v
        frame_flags ^= coded_flags
    }
    if(frame_flags&FLAG_STREAM_ID){
        stream_id                       v
    }
    if(frame_flags&FLAG_CODED_PTS){
        coded_pts                       v
    }
    if(frame_flags&FLAG_SIZE_MSB){
        data_size_msb                   v
    }
    if(frame_flags&FLAG_RESERVED)
        frame_res                       v
    for(i=0; i<frame_res; i++)
        reserved                        v
    if(frame_flags&FLAG_CHECKSUM){
        checksum                        u(32)
    }
    data

index:
    max_pts                             t
    syncpoints                          v
    for(i=0; i<syncpoints; i++){
        syncpoint_pos_div16             v
    }
    for(i=0; i<stream_count; i++){
        last_pts= -1
        for(j=0; j<syncpoints; ){
            x                           v
            type= x & 1
            x>>=1
            n=j
            if(type){
                flag= x & 1
                x>>=1
                while(x--)
                    has_keyframe[n++][i]=flag
                has_keyframe[n++][i]=!flag;
            }else{
                while(x != 1){
                    has_keyframe[n++][i]=x&1;
                    x>>=1;
                }
            }
            for(; j<n && j<syncpoints; j++){
                if (!has_keyframe[j][i]) continue
                A                           v
                if(!A){
                    A                       v
                    B                       v
                    eor_pts[j][i] = last_pts + A + B
                }else
                    B=0
                keyframe_pts[j][i] = last_pts + A
                last_pts += A + B
            }
        }
    }
    reserved_bytes
    index_ptr                           u(64)

info_packet:
    stream_id_plus1                     v
    chapter_id                          s (Note: Due to a typo this was v
                                           until 2006-11-04.)
    chapter_start                       t
    chapter_len                         v
    count                               v
    for(i=0; i<count; i++){
        name                            vb
        value                           s
        if (value==-1){
            type= "UTF-8"
            value                       vb
        }else if (value==-2){
            type                        vb
            value                       vb
        }else if (value==-3){
            type= "s"
            value                       s
        }else if (value==-4){
            type= "t"
            value                       t
        }else if (value<-4){
            type= "r"
            value.den= -value-4
            value.num                   s
        }else{
            type= "v"
        }
    }
    reserved_bytes

syncpoint:
    global_key_pts                      t
    back_ptr_div16                      v
    reserved_bytes

            Complete definition:


Tag description:
----------------

file_id_string
    "nut/multimedia container\0"
    The very first thing in every NUT file, useful for identifying NUT files.

*_startcode (f(64))
    all startcodes start with 'N'

main_startcode (f(64))
    0x7A561F5F04ADULL + (((uint64_t)('N'<<8) + 'M')<<48)

stream_startcode (f(64))
    0x11405BF2F9DBULL + (((uint64_t)('N'<<8) + 'S')<<48)

syncpoint_startcode (f(64))
    0xE4ADEECA4569ULL + (((uint64_t)('N'<<8) + 'K')<<48)

index_startcode (f(64))
    0xDD672F23E64EULL + (((uint64_t)('N'<<8) + 'X')<<48)

info_startcode (f(64))
    0xAB68B596BA78ULL + (((uint64_t)('N'<<8) + 'I')<<48)

version (v)
    NUT version. The current value is 3. All lower values are pre-freeze.

stream_count (v)
    number of streams in this file

time_base_count (v)
    number of different time bases in this file
    This MUST NOT be 0.

forward_ptr (v)
    Size of the packet data (exactly the distance from the first byte
    after the packet_header to the first byte of the next packet).
    Every NUT packet contains a forward_ptr immediately after its startcode
    with the exception of frame_code-based packets. The forward pointer
    can be used to skip over the packet without decoding its contents.

max_distance (v)
    maximum distance between startcodes. If p1 and p2 are the byte
    positions of the first byte of two consecutive startcodes, then
    p2-p1 MUST be less than or equal to max_distance unless the entire
    span from p1 to p2 comprises a single packet or a syncpoint
    followed by a single frame. This imposition places efficient upper
    bounds on seek operations and allows for the detection of damaged
    frame headers, should a chain of frame headers pass max_distance
    without encountering any startcode.

    Syncpoints SHOULD be placed immediately before a keyframe if the
    previous frame of the same stream was a non-keyframe, unless such
    non-keyframe - keyframe transitions are very frequent.

    SHOULD be set to <=32768.
    If the stored value is >65536 then max_distance MUST be set to 65536.

    This is also half the maximum frame size without a checksum after the
    frame header.


max_pts_distance (v)
    Maximum absolute difference of the pts of the new frame from last_pts in
    the timebase of the stream, without a checksum after the frame header.
    A frame header MUST include a checksum if abs(pts-last_pts) is
    strictly greater than max_pts_distance.
    Note that last_pts is not necessarily the pts of the last frame
    on the same stream, as it is altered by syncpoint timestamps.
    SHOULD NOT be higher than 1/timebase.

stream_id (v)
    Stream identifier
    stream_id MUST be < stream_count

stream_class (v)
    0    video
    1    audio
    2    subtitles
    3    userdata
    Note: The remaining values are reserved and MUST NOT be used.
          A demuxer MUST ignore streams with reserved classes.

fourcc (vb)
    identification for the codec
    example: "H264"
    MUST contain 2 or 4 bytes, note, this might be increased in the future
    if needed.
    The ID values used are the same as in AVI, so if a codec uses a specific
    FourCC in AVI then the same FourCC MUST be used here.

time_base_num (v) / time_base_denom (v) = time_base
    the length of a timer tick in seconds, this MUST be equal to the 1/fps
    if FLAG_FIXED_FPS is set
    time_base_num and time_base_denom MUST NOT be 0
    time_base_num and time_base_denom MUST be relatively prime
    time_base_denom MUST be < 2^31
    examples:
        fps       time_base_num    time_base_denom
        30        1                30
        29.97     1001             30000
        23.976    1001             24000
    There MUST NOT be 2 identical timebases in a file.
    There SHOULD NOT be more timebases than streams.

time_base_id (v)
    index into the time_base table
    MUST be < time_base_count.

convert_ts
    To switch from 2 different timebases, the following calculation is
    defined:

    ln        = from_time_base_num*to_time_base_denom
    sn        = from_timestamp
    d1        = from_time_base_denom
    d2        = to_time_base_num
    timestamp = (ln/d1*sn + ln%d1*sn/d1)/d2
    Note: This calculation MUST be done with unsigned 64 bit integers, and
    is equivalent to (ln*sn)/(d1*d2) but this would require a 96 bit integer.

compare_ts
    Compares timestamps from 2 different timebases,
    if a is before b then compare_ts(a, b) = -1
    if a is after  b then compare_ts(a, b) =  1
    else                  compare_ts(a, b) =  0

    Care must be taken that this is done exactly with no rounding errors,
    simply casting to float or double and doing the obvious
    a*timebase > b*timebase is not compliant or correct, neither is the
    same with integers, and
    a*a_timebase.num*b_timebase.den > b*b_timebase.num*a_timebase.den
    will overflow. One possible implementation which shouldn't overflow
    within the range of legal timestamps and timebases is:

    if (convert_ts(a, a_timebase, b_timebase) < b) return -1;
    if (convert_ts(b, b_timebase, a_timebase) < a) return  1;
    return 0;

msb_pts_shift (v)
    amount of bits in lsb_pts
    MUST be <16.

decode_delay (v)
    Size of the reordering buffer used to convert pts to dts.
    Codecs which do not support B-frames normally use 0.
    MPEG-1/MPEG-2-style codecs with B-frames use 1.
    H.264-style B-pyramid uses 2.
    H.264 and future codecs might need values >2.
    Audio codecs generally use 0. (We are not aware of any, but it
    is theoretically possible that a codec might need a value >0.)
    decode_delay MUST NOT be set higher than necessary for a codec.

stream_flags (v)
     Bit  Name            Description
       1  FLAG_FIXED_FPS  indicates that the fps is fixed

codec_specific_data (vb)
    Private global data for a codec (could be huffman tables or ...).
    If a codec has a global header it SHOULD be placed in here instead of
    at the start of every keyframe.
    The exact format is specified in the codec specification.
    For H.264 the NAL units MUST be formatted as in a bytestream
    (with 00 00 01 prefixes).
    codec_specific_data SHOULD contain exactly the essential global packets
    needed to decode a stream, more specifically it SHOULD NOT contain packets
    which contain only non essential metadata like author, title, ...
    It also MUST NOT contain normal packets which cause the reference decoder
    to generate any specific decoded samples.
    The encoder name and version shall be considered essential as it is very
    useful to work around possible encoder bugs.
    The global headers MUST consist of the normal
    sequence of header packets required for codec initialization, in the
    order defined in the codec spec. An implementation MAY strip metadata and
    other redundant information not necessary for correct playback from the
    global headers as long as no incorrect values are stored and as long as
    the stripped result is not less valid per codec spec as before stripping.

frame_code (f(8))
    frame_code is an 8-bit field which exists before every frame, it can
    store part of the size of the frame, the stream number, the timestamp
    and some flags amongst other things. What is not directly stored
    in it but is needed is stored in various fields immediately after it.
    The values stored in it can be found in the main header.
    The value 78 ('N') is forbidden to ensure that the byte is always
    different from the first byte of any startcode.
    A muxer SHOULD mark 0x00 and 0xFF as invalid to improve error
    detection.

flags[frame_code], frame_flags (v)
     Bit  Name             Description
       0  FLAG_KEY         If set, the frame is a keyframe.
       1  FLAG_EOR         If set, the stream has no relevance on
                           presentation. (EOR)
       3  FLAG_CODED_PTS   If set, coded_pts is in the frame header.
       4  FLAG_STREAM_ID   If set, stream_id is coded in the frame header.
       5  FLAG_SIZE_MSB    If set, data_size_msb at the frame header,
                           otherwise data_size_msb is 0.
       6  FLAG_CHECKSUM    If set, the frame header contains a checksum.
       7  FLAG_RESERVED    If set, reserved_count is coded in the frame header.
      12  FLAG_CODED       If set, coded_flags are stored in the frame header.
      13  FLAG_INVALID     If set, frame_code is invalid.

    EOR frames MUST be zero-length and must be set keyframe.
    All streams SHOULD end with EOR, where the pts of the EOR indicates the
    end presentation time of the final frame.
    An EOR set stream is unset by the first content frames.
    EOR can only be unset in streams with zero decode_delay .
    FLAG_CHECKSUM MUST be set if the frame's data_size is strictly greater than
    2*max_distance or the difference abs(pts-last_pts) is strictly greater than
    max_pts_distance (where pts represents this frame's pts and last_pts is
    defined as below).

last_pts
    The timestamp of the last frame with the same stream_id as the current.
    If there is no such frame between the last syncpoint and the current
    frame then the syncpoint timestamp is used, see global_key_pts.

stream_id[frame_code] (v)
    If FLAG_STREAM_ID is not set then this is the stream number for the
    frame following this frame_code.
    If FLAG_STREAM_ID is set then this value has no meaning.
    MUST be <250.

data_size_mul[frame_code] (v)
    If FLAG_SIZE_MSB is set then data_size_msb which is stored after the
    frame code is multiplied with it and forms the more significant part
    of the size of the following frame.
    If FLAG_SIZE_MSB is not set then this field has no meaning.
    MUST be <16384.

data_size_lsb[frame_code] (v)
    The less significant part of the size of the following frame.
    This added together with data_size_mul*data_size_msb is the size of
    the following frame.
    MUST be <16384.

pts_delta[frame_code] (s)
    If FLAG_CODED_PTS is set in the flags of the current frame then this
    value MUST be ignored, if FLAG_CODED_PTS is not set then pts_delta is the
    difference between the current pts and last_pts.
    MUST be <16384 and >-16384.

reserved_count[frame_code] (v)
    MUST be <256.

data_size
    The size of the following frame.
    data_size = data_size_lsb + data_size_msb * data_size_mul ;

coded_pts (v)
    If coded_pts < ( 1 << msb_pts_shift ) then it is an lsb
    pts, otherwise it is a full pts + ( 1 << msb_pts_shift ).
    lsb pts is converted to a full pts by:
    mask  = ( 1 << msb_pts_shift ) - 1;
    delta = last_pts - mask / 2
    pts   = ( (pts_lsb - delta) & mask ) + delta

lsb_pts
    Least significant bits of the pts in time_base precision.
        Example: IBBP display order
        keyframe pts=0                       -> pts=0
        frame                    lsb_pts=3   -> pts=3
        frame                    lsb_pts=1   -> pts=1
        frame