Regression - Memory leak when muxing raw h264 streams

[Francesco Bucciantini]

Summary of the bug:
When raw h264 streams (i.e raw_video.h264 files) are being muxed in a container, FFMpeg keeps allocating RAM with no clear progress until it goes into paging files on Windows or swap on Linux and then eventually crashes.

How to reproduce:

Step 1)
Create a simple Colorbars script with Avisynth and call it "AVS Script.avs"

ColorBars(1920, 1080, pixel_type="YUV422P16")

Step 2)
Encode the colorbars with x264 to create a raw_video.h264 stream

x264-10b.exe "AVS Script.avs" --preset medium --profile high422 --level 4.1 --opencl --keyint 1 --no-cabac --avcintra 100 --avcintra-flavor panasonic --deblock -1:-1 --tff --overscan show --colormatrix bt709 --range tv --log-level info --thread-input --transfer bt709 --colorprim bt709 --videoformat component --nal-hrd cbr --output-depth 10 --output-csp i422 --output "raw_video.h264"

Step 3)
Mux the newly produced raw_video.h264 into any container you like, be it mkv or mxf or whatever

ffmpeg.exe -hide_banner -i "raw_video.h264" -map 0:0 -c:v copy -f mxf "output.mxf"

Expected behavior:

FFMpeg muxes the file in mxf or whatever container you pick.

Actual behavior:

FFMpeg keeps allocating RAM until it crashes with no clear progress.

In the following example, I'm trying to use the newly encoded H264 stream and the audio from Avisynth on a real file to then re-encode the audio in PCM 24bit 48000Hz little endian and mux the whole thing in mxf

That's the latest master from today, namely N-114842-g639013aafc from the 17th of April 2024.
Using version N-97504-g1128aa8 from 2020 works instead with the exact same commands, which points to a regression.