MP4 demuxer breaks for edit-list and fragments

[Ulrik Mikaelsson]

Summary of the bug: When a fragmented mp4 contains an edit-list it is only applied to the first fragment. According to IEEE 14496-12, it should apply to "the movie and subsequent movie fragments".

The net effect is that all fragments after the first, are wrongly stamped with PTS ignoring the edit-list. For AAC, where priming-samples are encoded as an edit-list, this means some frames get the same PTS, and the second is dropped by the decoder, causing audible glitch. Furthermore, the subsequent fragments are all broken in A/V-sync.

How to reproduce:

# Boring sine-waves as input, but it audibly exposes the glitch in the first
#   segment change
% ffmpeg -f lavfi -i \
  "aevalsrc='(sin(sin(t)*8000)+sin(sin(t/1.4)*4000)+sin(sin(t/3)*8000))/3.1':s=48000:d=20" \
  -c:a aac flat.mp4

# Shaka packager produces fragmented MP4 with Edit-list to signal priming samples
% packager --fragment_duration=4 --segment_duration=4 \
  input=flat.mp4,stream=audio,format=mp4,out=frag.mp4

# Listen to "glitch" after 4 seconds
% ffplay frag.mp4

% ffprobe frag.mp4 -show_packets | grep pts
> ....
> pts=188416
> pts=189440
> pts=190464
> pts=191488
> pts=191488   # pts_time=3.989333
> pts=192512
> pts=193536
> pts=194560
> ...

For built-in AAC encoder, a single frame of priming samples will be time-duplicated and dropped. For libfdk_aac, two frames are duplicated and dropped. Consecutive fragments are then offset by one or two frames in PTS.

ffmpeg version 4.3.1 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 9.3.0 (Alpine 9.3.0)
configuration: --pkg-config-flags=--static --extra-cflags=-fopenmp --extra-ldflags='-static -fopenmp' --toolchain=hardened --disable-debug --disable-shared --disable-ffplay --enable-static --enable-gpl --enable-gray --enable-nonfree --enable-openssl --enable-iconv --enable-libxml2 --enable-libmp3lame --enable-libfdk-aac --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvpx --enable-libx264 --enable-libx265 --enable-libwebp --enable-libwavpack --enable-libspeex --enable-libaom --enable-libvidstab --enable-libkvazaar --enable-libfreetype --enable-fontconfig --enable-libfribidi --enable-libass --enable-libzimg --enable-libsoxr --enable-libopenjpeg --enable-libdav1d --enable-libxvid --enable-librav1e --enable-libsrt
libavutil      56. 51.100 / 56. 51.100
libavcodec     58. 91.100 / 58. 91.100
libavformat    58. 45.100 / 58. 45.100
libavdevice    58. 10.100 / 58. 10.100
libavfilter     7. 85.100 /  7. 85.100
libswscale      5.  7.100 /  5.  7.100
libswresample   3.  7.100 /  3.  7.100
libpostproc    55.  7.100 / 55.  7.100

[Balling]

Does the same happen to eac3? First 256 samples are dropped there with editlist.

[Ulrik Mikaelsson]

Surprisingly, it seems to not happen for eac3. Not sure why not.

[Ulrik Mikaelsson]

ah Given EAC-3 having much bigger frame-size than priming samples, no packets are dropped, but there is a PTS discontinuity:

% ffprobe -show_packets frag.mp4 | grep pts= | gawk -F= '/pts=/{print($2-last); last=$2}'
...
1536
1536
1536
1280
1536
1536
...

[Balling]

I will also point out that neither aac nor eac3 are dropping ramainder garbage/silence samples!! Wow. AAC encoder does apply correct editlist media duration WHICH IS NOT being applied on decode! Known bug fixed in Chrome that uses ffmpeg! While EAC3 has outright WRONG media duration in editlist with included 512 samples of garbage ramainder, that should have been edited out!