Bad decoding of MP4 containing differently encoded videos joined

[Anatoliy]

Summary of the bug:
What I trying to accomplish: I want to concatenate multiple almost independent, but having the same format (resolution, codec, pix_fmt and all others), chunks of video, size of which is in a range 1-25.
The problem I encountered: Resulting video cannot be decoded (at least) by libav* and players that depend on it (mpv, vlc).
The exact command line you used:

#!/usr/bin/env bash

set -xueo pipefail

rm -f {1,2,12}.mp4 ffmpeg-*.log
./ffmpeg -f lavfi -i testsrc=s=640x480 -frames:v 1 -codec:v libx265 -x265-params keyint=1:min-keyint=1 1.mp4
./ffmpeg -f lavfi -i testsrc=s=640x480 -frames:v 2 -codec:v libx265 2.mp4
cat >cat.txt <<_EOF
file 1.mp4
file 2.mp4
_EOF
./ffmpeg -safe 0 -f concat -i cat.txt -codec:v copy 12.mp4
./ffmpeg -v 9 -loglevel 99 -report -i 12.mp4 -f null - &>err.txt

The full uncut console output:

ffmpeg started on 2024-08-16 at 18:53:39
Report written to "ffmpeg-20240816-185339.log"
Log level: 99
ffmpeg version 7.0.1-static https://johnvansickle.com/ffmpeg/  Copyright (c) 2000-2024 the FFmpeg developers
  built with gcc 8 (Debian 8.3.0-6)
  configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gmp --enable-libgme --enable-gray --enable-libaom --enable-libfribidi --enable-libass --enable-libvmaf --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libdav1d --enable-libxvid --enable-libzvbi --enable-libzimg
  libavutil      59.  8.100 / 59.  8.100
  libavcodec     61.  3.100 / 61.  3.100
  libavformat    61.  1.100 / 61.  1.100
  libavdevice    61.  1.100 / 61.  1.100
  libavfilter    10.  1.100 / 10.  1.100
  libswscale      8.  1.100 /  8.  1.100
  libswresample   5.  1.100 /  5.  1.100
  libpostproc    58.  1.100 / 58.  1.100
Splitting the commandline.
Reading option '-v' ... matched as option 'v' (set logging level) with argument '9'.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument '99'.
Reading option '-report' ... matched as option 'report' (generate a report) with argument '1'.
Reading option '-i' ... matched as input url with argument '12.mp4'.
Reading option '-f' ... matched as option 'f' (force container format (auto-detected otherwise)) with argument 'null'.
Reading option '-' ... matched as output url.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option v (set logging level) with argument 9.
Applying option report (generate a report) with argument 1.
Successfully parsed a group of options.
Parsing a group of options: input url 12.mp4.
Successfully parsed a group of options.
Opening an input file: 12.mp4.
[AVFormatContext @ 0x77db140] Opening '12.mp4' for reading
[file @ 0x77db940] Setting default whitelist 'file,crypto,data'
Probing mov,mp4,m4a,3gp,3g2,mj2 score:100 size:2048
Probing mp3 score:1 size:2048
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] Format mov,mp4,m4a,3gp,3g2,mj2 probed with size=2048 and score=100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'ftyp' parent:'root' sz: 28 8 16012
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] ISO: File Type Major Brand: isom
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'free' parent:'root' sz: 8 36 16012
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'mdat' parent:'root' sz: 12785 44 16012
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'moov' parent:'root' sz: 3191 12829 16012
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'mvhd' parent:'moov' sz: 108 8 3183
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] time scale = 1000
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'trak' parent:'moov' sz: 2978 116 3183
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'tkhd' parent:'trak' sz: 92 8 2970
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'edts' parent:'trak' sz: 36 100 2970
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'elst' parent:'edts' sz: 28 8 28
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] track[0].edit_count = 1
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] duration=120 time=0 rate=1.000000
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'mdia' parent:'trak' sz: 2842 136 2970
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'mdhd' parent:'mdia' sz: 32 8 2834
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'hdlr' parent:'mdia' sz: 45 40 2834
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] ctype=[0][0][0][0]
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] stype=vide
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'minf' parent:'mdia' sz: 2757 85 2834
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'vmhd' parent:'minf' sz: 20 8 2749
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'dinf' parent:'minf' sz: 36 28 2749
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'dref' parent:'dinf' sz: 28 8 28
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] Unknown dref type 0x206c7275 size 12
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'stbl' parent:'minf' sz: 2693 64 2749
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'stsd' parent:'stbl' sz: 2557 8 2685
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] size=2541 4CC=hev1 codec_type=0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'hvcC' parent:'stsd' sz: 2409 8 2455
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'fiel' parent:'stsd' sz: 10 2417 2455
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'pasp' parent:'stsd' sz: 16 2427 2455
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'btrt' parent:'stsd' sz: 20 2443 2455
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'stts' parent:'stbl' sz: 24 2565 2685
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] track[0].stts.entries = 1
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] sample_count=3, sample_duration=512
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'stss' parent:'stbl' sz: 24 2589 2685
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] keyframe_count = 2
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'stsc' parent:'stbl' sz: 28 2613 2685
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] track[0].stsc.entries = 1
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'stsz' parent:'stbl' sz: 32 2641 2685
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] sample_size = 0 sample_count = 3
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'stco' parent:'stbl' sz: 20 2673 2685
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] AVIndex stream 0, sample 0, offset 2c, dts 0, size 8185, distance 0, keyframe 1
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] AVIndex stream 0, sample 1, offset 2025, dts 512, size 4458, distance 0, keyframe 1
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] AVIndex stream 0, sample 2, offset 318f, dts 1024, size 134, distance 1, keyframe 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] Processing st: 0, edit list 0 - media time: 0, duration: 1536
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'udta' parent:'moov' sz: 97 3094 3183
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'meta' parent:'udta' sz: 89 8 89
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'hdlr' parent:'meta' sz: 33 8 77
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] ctype=[0][0][0][0]
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] stype=mdir
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'ilst' parent:'meta' sz: 44 41 77
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] type:'[169]too' parent:'ilst' sz: 36 8 36
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] on_parse_exit_offset=16012
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] Before avformat_find_stream_info() pos: 16012 bytes read:16012 seeks:0 nb_streams:1
[hevc @ 0x77dc700] nal_unit_type: 32(VPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x77dc700] Decoding VPS
[hevc @ 0x77dc700] Range Extension profile bitstream
[hevc @ 0x77dc700] nal_unit_type: 33(SPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x77dc700] Decoding SPS
[hevc @ 0x77dc700] Range Extension profile bitstream
[hevc @ 0x77dc700] Decoding VUI
[hevc @ 0x77dc700] nal_unit_type: 34(PPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x77dc700] Decoding PPS
[hevc @ 0x77dc700] nal_unit_type: 39(SEI_PREFIX), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x77dc700] Decoding SEI
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] stream 0, sample 0, dts 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] All info found
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] stream 0: start_time: 0 duration: 0.12
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] format: start_time: 0 duration: 0.12 (estimate from stream) bitrate=1067 kb/s
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] After avformat_find_stream_info() pos: 8229 bytes read:16012 seeks:0 frames:1
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '12.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2mp41
    encoder         : Lavf61.1.100
  Duration: 00:00:00.12, start: 0.000000, bitrate: 1067 kb/s
  Stream #0:0[0x1](und), 1, 1/12800: Video: hevc (Rext), 1 reference frame (hev1 / 0x31766568), gbrp(pc, gbr/unknown/unknown, progressive), 640x480 [SAR 1:1 DAR 4:3], 0/1, 851 kb/s, 25 fps, 25 tbr, 12800 tbn (default)
      Metadata:
        handler_name    : VideoHandler
        vendor_id       : [0][0][0][0]
        encoder         : Lavc61.3.100 libx265
Successfully opened the file.
Parsing a group of options: output url -.
Applying option f (force container format (auto-detected otherwise)) with argument null.
Successfully parsed a group of options.
Opening an output file: -.
[out#0/null @ 0x77e3a40] No explicit maps, mapping streams automatically...
[vost#0:0/wrapped_avframe @ 0x780e7c0] Created video stream from input stream 0:0
detected 20 logical cores
[hevc @ 0x7811cc0] nal_unit_type: 32(VPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x7811cc0] Decoding VPS
[hevc @ 0x7811cc0] Range Extension profile bitstream
[hevc @ 0x7811cc0] nal_unit_type: 33(SPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x7811cc0] Decoding SPS
[hevc @ 0x7811cc0] Range Extension profile bitstream
[hevc @ 0x7811cc0] Decoding VUI
[hevc @ 0x7811cc0] nal_unit_type: 34(PPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x7811cc0] Decoding PPS
[hevc @ 0x7811cc0] nal_unit_type: 39(SEI_PREFIX), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x7811cc0] Decoding SEI
Successfully opened the file.
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (native) -> wrapped_avframe (native))
[vost#0:0/wrapped_avframe @ 0x780e7c0] Starting thread...
[vf#0:0 @ 0x780cc40] Starting thread...
[vist#0:0/hevc @ 0x7821e40] [dec:hevc @ 0x7810a00] Starting thread...
[in#0/mov,mp4,m4a,3gp,3g2,mj2 @ 0x77dae40] Starting thread...
Press [q] to stop, [?] for help
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] stream 0, sample 1, dts 40000
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x77db140] stream 0, sample 2, dts 80000
[in#0/mov,mp4,m4a,3gp,3g2,mj2 @ 0x77dae40] EOF while reading input
[in#0/mov,mp4,m4a,3gp,3g2,mj2 @ 0x77dae40] Terminating thread with return code 0 (success)
[hevc @ 0x7811cc0] nal_unit_type: 32(VPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x7811cc0] nal_unit_type: 33(SPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x7811cc0] nal_unit_type: 34(PPS), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x7811cc0] nal_unit_type: 39(SEI_PREFIX), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x7811cc0] nal_unit_type: 20(IDR_N_LP), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x7811cc0] Decoding VPS
[hevc @ 0x7811cc0] Range Extension profile bitstream
[hevc @ 0x7811cc0] Decoding SPS
[hevc @ 0x7811cc0] Range Extension profile bitstream
[hevc @ 0x7811cc0] Decoding VUI
[hevc @ 0x7811cc0] Decoding PPS
[hevc @ 0x7811cc0] Decoding SEI
[hevc @ 0x7811cc0] Format gbrp chosen by get_format().
[hevc @ 0x7811cc0] Output frame with POC 0.
[hevc @ 0x780b6c0] nal_unit_type: 20(IDR_N_LP), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x780b6c0] Output frame with POC 0.
[vist#0:0/hevc @ 0x7821e40] [dec:hevc @ 0x7810a00] Decoder thread received EOF packet
[hevc @ 0x7a91ac0] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0, temporal_id: 0
[hevc @ 0x7a91ac0] Invalid number of merging MVP candidates: 254.
[hevc @ 0x7a91ac0] Error parsing NAL unit #0.
[hevc @ 0x7811cc0] Decoded frame with POC 0.
[graph 0 input from stream 0:0 @ 0x7fd5fc003400] Setting 'video_size' to value '640x480'
[graph 0 input from stream 0:0 @ 0x7fd5fc003400] Setting 'pix_fmt' to value '71'
[graph 0 input from stream 0:0 @ 0x7fd5fc003400] Setting 'time_base' to value '1/12800'
[graph 0 input from stream 0:0 @ 0x7fd5fc003400] Setting 'pixel_aspect' to value '1/1'
[graph 0 input from stream 0:0 @ 0x7fd5fc003400] Setting 'colorspace' to value '0'
[graph 0 input from stream 0:0 @ 0x7fd5fc003400] Setting 'range' to value '2'
[graph 0 input from stream 0:0 @ 0x7fd5fc003400] Setting 'frame_rate' to value '25/1'
[graph 0 input from stream 0:0 @ 0x7fd5fc003400] w:640 h:480 pixfmt:gbrp tb:1/12800 fr:25/1 sar:1/1 csp:gbr range:pc
[hevc @ 0x780b6c0] Decoded frame with POC 0.
[AVFilterGraph @ 0x7fd5fc0011c0] query_formats: 3 queried, 6 merged, 0 already done, 0 delayed
[graph 0 input from stream 0:0 @ 0x7fd5fc003400] video frame properties congruent with link at pts_time: 0
Output #0, null, to 'pipe:':
  Metadata:
    major_brand     : isomDecoding error: Invalid data found when processing input

    minor_version   : 512
    compatible_brands: isomiso2mp41
    encoder         : Lavf61.1.100
  Stream #0:0(und), 0, 1/25: Video: wrapped_avframe, 1 reference frame, gbrp(pc, gbr/unknown/unknown, progressive), 640x480 [SAR 1:1 DAR 4:3], 0/1, q=2-31, 200 kb/s, 25 fps, 25 tbn (default)
      Metadata:
        handler_name    : VideoHandler
        vendor_id       : [0][0][0][0]
        encoder         : Lavc61.3.100 wrapped_avframe
[out#0/null @ 0x77e3a40] Starting thread...
[vist#0:0/hevc @ 0x7821e40] [dec:hevc @ 0x7810a00] Decoder returned EOF, finishing
[vist#0:0/hevc @ 0x7821e40] [dec:hevc @ 0x7810a00] Terminating thread with return code 0 (success)
[out_0_0 @ 0x7fd5fc004b80] EOF on sink link out_0_0:default.
[vf#0:0 @ 0x780cc40] Filtergraph returned EOF, finishing
[vf#0:0 @ 0x780cc40] All consumers returned EOF
[vost#0:0/wrapped_avframe @ 0x780e7c0] Encoder thread received EOF
[vost#0:0/wrapped_avframe @ 0x780e7c0] Terminating thread with return code 0 (success)
[out#0/null @ 0x77e3a40] All streams finished
[out#0/null @ 0x77e3a40] Terminating thread with return code 0 (success)
[vf#0:0 @ 0x780cc40] Terminating thread with return code 0 (success)
[out#0/null @ 0x77e3a40] Output file #0 (pipe:):
[out#0/null @ 0x77e3a40]   Output stream #0:0 (video): 2 frames encoded; 2 packets muxed (880 bytes); 
[out#0/null @ 0x77e3a40]   Total: 2 packets (880 bytes) muxed
[out#0/null @ 0x77e3a40] video:1KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: unknown
frame=    2 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.08 bitrate=N/A speed=24.3x    
[in#0/mov,mp4,m4a,3gp,3g2,mj2 @ 0x77dae40] Input file #0 (12.mp4):
[in#0/mov,mp4,m4a,3gp,3g2,mj2 @ 0x77dae40]   Input stream #0:0 (video): 3 packets read (12777 bytes); 2 frames decoded; 1 decode errors; 
[in#0/mov,mp4,m4a,3gp,3g2,mj2 @ 0x77dae40]   Total: 3 packets (12777 bytes) demuxed
[AVIOContext @ 0x77e3d80] Statistics: 16012 bytes read, 0 seeks

-v error only:

[hevc @ 0x77fab80] Invalid number of merging MVP candidates: 254.
[vist#0:0/hevc @ 0x758afc0] [dec:hevc @ 0x757a100] Decoding error: Invalid data found when processing input

[Anatoliy]

ffmpeg-20240816-185339.log

[MasterQuestionable]

͏    Refer: ​https://trac.ffmpeg.org/ticket/11056#comment:11

[MasterQuestionable]

͏    Not the same. Mistake.

[MasterQuestionable]

͏    The command indicated concatenation of differently encoded videos:
͏    Which most containers wouldn't properly support.

͏    Note different encoding settings of the same codec, maybe no different than different codecs outright.
͏    Woefullest case: different videos of different resolutions...
͏    No sensible general handling (without involve express designation) possible.

[Anatoliy]

The difference only is between 1-frame chunks and non-1-frame chunks. 1-frame chunks are special in this regard.
Seems this is the field of undefined behavior (but I suspect it is very close to become defined, just need some improvement to handle the case (because for x264 there is no such a problem), so this should not be considered as a bug, but as a feature request though).

[MasterQuestionable]

͏    Different "keyint" settings is more than that.
͏    It's up to the decoder to decide how shall handle alike.
͏    However this is somewhat "Codec container wrestling each other..."

͏    See also:
͏    ​https://trac.ffmpeg.org/ticket/5309#comment:16
͏    ​https://trac.ffmpeg.org/ticket/11110#comment:3

[Anatoliy]

Thank you for the links.
For those who also want to encode video as single-GOP chukns (for me it is needed to encode atlases for VR video) there is a solution: to use keyint=-1.

[MasterQuestionable]

͏    "keyint=infinite" preferable: which should also work for x264. ("keyint=-1" wouldn't)

͏    Note this doesn't mean no further I-frame later:
͏    Shall be still inserted when appropriate?
͏    (haven't detailedly verified, too many things cryptic with these codecs...)

[Anatoliy]

From x265
Invalid value for keyint: infinite.
Invalid value for keyint: inf.

[MasterQuestionable]

͏    Bravo in-compat...
͏    Makes me wondrous: ​https://trac.ffmpeg.org/ticket/11155