aac encoder adds extra frame(s)

[slhck]

Summary of the bug: The aac encoder adds one extra audio frame when encoding

How to reproduce:

ffmpeg -i out.mp4 -c:v copy -c:a aac out2.mp4
ffmpeg version N-78077-g5989add-tessus Copyright (c) 2000-2016 the FFmpeg developers
  built with Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn)
  configuration: --cc=/usr/bin/clang --prefix=/opt/ffmpeg --as=yasm --extra-version=tessus --enable-avisynth --enable-fontconfig --enable-gpl --enable-libass --enable-libbluray --enable-libfreetype --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzmq --enable-version3 --disable-ffplay --disable-indev=qtkit --disable-indev=x11grab_xcb
  libavutil      55. 13.100 / 55. 13.100
  libavcodec     57. 22.100 / 57. 22.100
  libavformat    57. 21.101 / 57. 21.101
  %                                                                                                                                                       !3491
  libavdevice    57.  0.100 / 57.  0.100
  libavfilter     6. 25.100 /  6. 25.100
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.101 /  2.  0.101
  libpostproc    54.  0.100 / 54.  0.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'out.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf56.40.101
  Duration: 00:12:53.97, start: 0.000000, bitrate: 2180 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080, 1857 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 317 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
[mp4 @ 0x7f8f8b842a00] Codec for stream 0 does not use global headers but container format requires global headers
Output #0, mp4, to 'out2.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.21.101
    Stream #0:0(eng): Video: h264 ([33][0][0][0] / 0x0021), yuv420p, 1920x1080, q=2-31, 1857 kb/s, 25 fps, 25 tbr, 12800 tbn, 12800 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(eng): Audio: aac (LC) ([64][0][0][0] / 0x0040), 48000 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
      encoder         : Lavc57.22.100 aac
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
frame=19348 fps=353 q=-1.0 Lsize=  188726kB time=00:12:53.97 bitrate=1997.5kbits/s speed=14.1x
video:175481kB audio:12669kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.306213%
[aac @ 0x7f8f8b806600] Qavg: 691.497

The input file:

ffprobe -i out.mp4 -show_streams | grep nb_frames
…
nb_frames=36280

The output file has one extra (audio) frame:

ffprobe -i out2.mp4 -show_streams | grep nb_frames
…
nb_frames=36281

This means that if you re-encode a file multiple times, audio and video will become more and more asynchronous.

The same bug affects libfdk_aac as well, although it adds two frames. When I add -shortest, I get one frame less than the original file.

[Carl Eugen Hoyos]

Regression since 151ecc2a / 89eea6df

I can reproduce the issue with the following command lines:

$ ffmpeg -f lavfi -i sine=d=1 out.aac
$ ffmpeg -i out.aac out2.aac
$ ffmpeg -i out2.aac -acodec copy -map 0 -f image2 out%2d

[Hendrik]

This is normal and expected behavior of any AAC encoder. It produces "preroll" data that the decoder is supposed to skip.

[slhck]

Replying to heleppkes:

This is normal and expected behavior of any AAC encoder. It produces "preroll" data that the decoder is supposed to skip.

How does the decoder know it's pre-roll data? Like I mentioned, if you re-encode the file multiple times it's not skipped, but added up.

[Hendrik]

The encoder sets avctx->initial_padding, and the muxers are supposed to offset the timestamps in the file appropriately, so while yes it may get longer, it should not cause sync issues.

If this is a bug, its not a bug in aacenc.

[Carl Eugen Hoyos]

How is the adts muxer supposed to set preroll in the output file?

[Hendrik]

raw files don't have timestamps, so all bets are off there.

[Hendrik]

To be precise, the encoder sets an initial_padding and produces one frame with negative timestamps. *ideally* the decoder would skip these initial samples after decoding, however unless a container has a way to signal how much padding exists... it wouldnt know how much.

[Carl Eugen Hoyos]

Isn't it a bug that ffmpeg -i input.aac output.aac makes the output file longer (and longer). Or is this a property of aac?

[Hendrik]

Its not a bug since raw aac files just don't have any way to signal how much data is to be skipped. Can't be fixed.

So yes, its a property of aac that it needs preroll data. Advanced containers could have ways to signal how much preroll exists and allow skipping it, but raw adts files of course don't have such metadata.

[slhck]

MP4 is not an advanced container then…?

Not sure how to work around this, I'm afraid.

[Hendrik]

Just adding one more audio frame doesn't really harm, because as I said it'll offset the timestamps and maintain A/V sync - and you shouldn't be re-encoding in a loop to add more and more frames in any case. ;)

[slhck]

I have a client who says they're experiencing a problem when encoding multiple times — I know it shouldn't be done for quality reasons, but the A/V sync is still lost.