The v4l2m2m decoder leaks memory: avpkt in v4l2_receive_frame is never freed

[Matrin Dørum]

I've just spent a while in valgrind and GDB trying to figure out why my video decoding program is leaking memory when using the h264_v4l2m2m decoder, but not when using the regular h264 decoder.

In v4l2_receive_frame in libavcodec/v4l2_m2m_dec.c, the decoder gets a packet from ff_decode_get_packet, but the packet is never freed. Its refcount stays at 1 forever. I know this is a bug with FFmpeg and not with my code, because ffmpeg CLI tool acts the same way; valgrind ffmpeg -i input-file.h264 output-file.h264 reports 0 lost bytes, while valgrind ffmpeg -c:v h264_v4l2m2m -i input-file.h264 output-file.h264 reports a bunch of leaked memory.

I'm running this on a DART-SD410 (with a Qualcomm Snapdragon 410 and a Qualcomm Venus decoder), with Linux 4.9.39+linaro. Here's the output of v4l2-ctl:

$ v4l2-ctl --all -d /dev/video4

Driver Info (not using libv4l2):
        Driver name   : qcom-venus
        Card type     : Qualcomm Venus video decoder
        Bus info      : platform:qcom-venus
        Driver version: 4.9.39
        Capabilities  : 0x84204000
                Video Memory-to-Memory Multiplanar
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps   : 0x04204000
                Video Memory-to-Memory Multiplanar
                Streaming
                Extended Pix Format
Priority: 2
Format Video Capture Multiplanar:
        Width/Height      : 1280/736
        Pixel Format      : 'NV12'
        Field             : None
        Number of planes  : 1
        Flags             :
        Colorspace        : Default
        Transfer Function : Default
        YCbCr Encoding    : Default
        Quantization      : Default
        Plane 0           :
           Bytes per Line : 1280
           Size Image     : 1425408
Format Video Output Multiplanar:
        Width/Height      : 1280/720
        Pixel Format      : 'H264'
        Field             : None
        Number of planes  : 1
        Flags             :
        Colorspace        : Default
        Transfer Function : Default
        YCbCr Encoding    : Default
        Quantization      : Default
        Plane 0           :
           Bytes per Line : 0
           Size Image     : 691328

Codec Controls

       mpeg4_loop_filter_enable (bool)   : default=0 value=0
                     h264_level (menu)   : min=0 max=15 default=0 value=0 flags=volatile
                   h264_profile (menu)   : min=0 max=16 default=0 value=0 flags=volatile
                    mpeg4_level (menu)   : min=0 max=7 default=0 value=0 flags=volatile
                  mpeg4_profile (menu)   : min=0 max=4 default=0 value=0 flags=volatile
                    vpx_profile (int)    : min=0 max=3 step=1 default=0 value=0 flags=volatile

Here's the full output of valgrind and ffmpeg:

$ valgrind --leak-check=full ffmpeg -v 9 -loglevel 99 -c:v h264_v4l2m2m -i test-320x240.h264 test-output.h264

==18427== Memcheck, a memory error detector
==18427== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==18427== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==18427== Command: ffmpeg -v 9 -loglevel 99 -c:v h264_v4l2m2m -i test-320x240.h264 test-output.h264
==18427==
ffmpeg version 4.0.1 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.9) 20160609
  configuration: --enable-cross-compile --cross-prefix=aarch64-linux-gnu- --arch=aarch64 --target-os=linux --enable-debug --enable-shared --disable-stripping --disable-doc --prefix=//mnt/inst
  libavutil      56. 14.100 / 56. 14.100
  libavcodec     58. 18.100 / 58. 18.100
  libavformat    58. 12.100 / 58. 12.100
  libavdevice    58.  3.100 / 58.  3.100
  libavfilter     7. 16.100 /  7. 16.100
  libswscale      5.  1.100 /  5.  1.100
  libswresample   3.  1.100 /  3.  1.100
Splitting the commandline.
Reading option '-v' ... matched as option 'v' (set logging level) with argument '9'.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument '99'.
Reading option '-c:v' ... matched as option 'c' (codec name) with argument 'h264_v4l2m2m'.
Reading option '-i' ... matched as input url with argument 'test-320x240.h264'.
Reading option 'test-output.h264' ... matched as output url.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option v (set logging level) with argument 9.
Successfully parsed a group of options.
Parsing a group of options: input url test-320x240.h264.
Applying option c:v (codec name) with argument h264_v4l2m2m.
Successfully parsed a group of options.
Opening an input file: test-320x240.h264.
[NULL @ 0x61667b0] Opening 'test-320x240.h264' for reading
[file @ 0x6167440] Setting default whitelist 'file,crypto'
Probing h264 score:51 size:2048
Probing mp3 score:1 size:2048
[h264 @ 0x61667b0] Format h264 probed with size=2048 and score=51
[h264 @ 0x61667b0] Before avformat_find_stream_info() pos: 0 bytes read:32768 seeks:0 nb_streams:1
[AVBSFContext @ 0x61d0e30] nal_unit_type: 9, nal_ref_idc: 0
[AVBSFContext @ 0x61d0e30] nal_unit_type: 7, nal_ref_idc: 3
[AVBSFContext @ 0x61d0e30] nal_unit_type: 8, nal_ref_idc: 3
[AVBSFContext @ 0x61d0e30] nal_unit_type: 6, nal_ref_idc: 0
[AVBSFContext @ 0x61d0e30] nal_unit_type: 5, nal_ref_idc: 3
[h264 @ 0x6179480] nal_unit_type: 9, nal_ref_idc: 0
[h264 @ 0x6179480] nal_unit_type: 7, nal_ref_idc: 3
[h264 @ 0x6179480] nal_unit_type: 8, nal_ref_idc: 3
[h264 @ 0x6179480] nal_unit_type: 6, nal_ref_idc: 0
[h264 @ 0x6179480] nal_unit_type: 5, nal_ref_idc: 3
[h264 @ 0x6179480] Format yuv444p chosen by get_format().
[h264 @ 0x6179480] Reinit context to 320x240, pix_fmt: yuv444p
[h264 @ 0x6179480] no picture
[h264 @ 0x61667b0] stream 0: start_time: -7686143364045.646 duration: -7686143364045.646
[h264 @ 0x61667b0] format: start_time: -9223372036854.775 duration: -9223372036854.775 bitrate=0 kb/s
[h264 @ 0x61667b0] After avformat_find_stream_info() pos: 641770 bytes read:641770 seeks:0 frames:105
Input #0, h264, from 'test-320x240.h264':
  Duration: N/A, bitrate: N/A
    Stream #0:0, 105, 1/1200000: Video: h264 (High 4:4:4 Predictive), 1 reference frame, yuv444p(tv, bt470bg/smpte170m/bt709, progressive, left), 320x240 [SAR 1:1 DAR 4:3], 0/1, 30 fps, 30 tbr, 1200k tbn, 60 tbc
Successfully opened the file.
Parsing a group of options: output url test-output.h264.
Successfully parsed a group of options.
Opening an output file: test-output.h264.
[file @ 0x64eb6a0] Setting default whitelist 'file,crypto'
Successfully opened the file.
[h264_v4l2m2m @ 0x64e7cf0] probing device /dev/video0
[h264_v4l2m2m @ 0x64e7cf0] driver 'qcom-camss' on card 'Qualcomm Camera Subsystem'
[h264_v4l2m2m @ 0x64e7cf0] v4l2 output format not supported
[h264_v4l2m2m @ 0x64e7cf0] probing device /dev/video5
[h264_v4l2m2m @ 0x64e7cf0] driver 'qcom-venus' on card 'Qualcomm Venus video encoder'
[h264_v4l2m2m @ 0x64e7cf0] v4l2 output format not supported
[h264_v4l2m2m @ 0x64e7cf0] probing device /dev/video4
[h264_v4l2m2m @ 0x64e7cf0] driver 'qcom-venus' on card 'Qualcomm Venus video decoder'
[h264_v4l2m2m @ 0x64e7cf0] Using device /dev/video4
[h264_v4l2m2m @ 0x64e7cf0] driver 'qcom-venus' on card 'Qualcomm Venus video decoder'
[h264_v4l2m2m @ 0x64e7cf0] output : H264 16 buffers initialized: 0320x0240, sizeimage 00057728, bytesperline 00000000
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (h264_v4l2m2m) -> h264 (h264_v4l2m2m))
Press [q] to stop, [?] for help
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
[AVBSFContext @ 0x6539eb0] The input looks like it is Annex B already
==18427== Syscall param ioctl(VKI_V4L2_S_SELECTION) points to uninitialised byte(s)
==18427==    at 0x60D891C: ioctl (ioctl.S:26)
==18427==    by 0x525FD3B: v4l2_try_start (v4l2_m2m_dec.c:72)
==18427==    by 0x525FD3B: v4l2_receive_frame (v4l2_m2m_dec.c:151)
==18427==    by 0x4E763C3: decode_receive_frame_internal (decode.c:610)
==18427==    by 0x4E78EAF: avcodec_send_packet (decode.c:674)
==18427==    by 0x42A447: decode (ffmpeg.c:2224)
==18427==    by 0x42A447: decode_video (ffmpeg.c:2368)
==18427==    by 0x42B213: process_input_packet (ffmpeg.c:2609)
==18427==    by 0x409EEB: process_input (ffmpeg.c:4445)
==18427==    by 0x409EEB: transcode_step (ffmpeg.c:4565)
==18427==    by 0x409EEB: transcode (ffmpeg.c:4619)
==18427==    by 0x409EEB: main (ffmpeg.c:4819)
==18427==  Address 0x1ffefffcdc is on thread 1's stack
==18427==  in frame #1, created by v4l2_receive_frame (v4l2_m2m_dec.c:129)
==18427==
[h264_v4l2m2m @ 0x64e7cf0] capture: NV12 20 buffers initialized: 0320x0256, sizeimage 00159744, bytesperline 00000384
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
detected 4 logical cores
[graph 0 input from stream 0:0 @ 0x653f760] Setting 'video_size' to value '320x240'
[graph 0 input from stream 0:0 @ 0x653f760] Setting 'pix_fmt' to value '23'
[graph 0 input from stream 0:0 @ 0x653f760] Setting 'time_base' to value '1/1200000'
[graph 0 input from stream 0:0 @ 0x653f760] Setting 'pixel_aspect' to value '0/1'
[graph 0 input from stream 0:0 @ 0x653f760] Setting 'sws_param' to value 'flags=2'
[graph 0 input from stream 0:0 @ 0x653f760] Setting 'frame_rate' to value '30/1'
[graph 0 input from stream 0:0 @ 0x653f760] w:320 h:240 pixfmt:nv12 tb:1/1200000 fr:30/1 sar:0/1 sws_param:flags=2
[AVFilterGraph @ 0x653e7e0] query_formats: 3 queried, 2 merged, 0 already done, 0 delayed
[h264_v4l2m2m @ 0x64ea350] probing device /dev/video0
[h264_v4l2m2m @ 0x64ea350] driver 'qcom-camss' on card 'Qualcomm Camera Subsystem'
[h264_v4l2m2m @ 0x64ea350] v4l2 output format not supported
[h264_v4l2m2m @ 0x64ea350] probing device /dev/video5
[h264_v4l2m2m @ 0x64ea350] driver 'qcom-venus' on card 'Qualcomm Venus video encoder'
[h264_v4l2m2m @ 0x64ea350] Using device /dev/video5
[h264_v4l2m2m @ 0x64ea350] driver 'qcom-venus' on card 'Qualcomm Venus video encoder'
[h264_v4l2m2m @ 0x64ea350] output : NV12 16 buffers initialized: 0320x0256, sizeimage 00159744, bytesperline 00000384
[h264_v4l2m2m @ 0x64ea350] capture: H264 04 buffers initialized: 0320x0256, sizeimage 00061440, bytesperline 00000000
[h264_v4l2m2m @ 0x64ea350] Encoder: number of B-frames = 0
[h264_v4l2m2m @ 0x64ea350] Encoder: header mode = 0
[h264_v4l2m2m @ 0x64ea350] Encoder: bit rate = 200000
[h264_v4l2m2m @ 0x64ea350] Encoder: gop size = 12
[h264_v4l2m2m @ 0x64ea350] Encoder Context: id (27), profile (-99), frame rate(30/1), number b-frames (0), gop size (12), bit rate (200000), qmin (2), qmax (31)
[h264_v4l2m2m @ 0x64ea350] h264 profile not found
[h264_v4l2m2m @ 0x64ea350] Encoder adjusted: qmin (0), qmax (51)
[h264_v4l2m2m @ 0x64ea350] Encoder: minimum video quantizer scale = 0
[h264_v4l2m2m @ 0x64ea350] Encoder: maximum video quantizer scale = 51
Output #0, h264, to 'test-output.h264':
  Metadata:
    encoder         : Lavf58.12.100
    Stream #0:0, 0, 1/30: Video: h264 (h264_v4l2m2m), 1 reference frame, nv12(left), 320x240, 0/1, q=2-31, 200 kb/s, 30 fps, 30 tbn, 30 tbc
    Metadata:
      encoder         : Lavc58.18.100 h264_v4l2m2m
Clipping frame in rate conversion by 0.000008
cur_dts is invalid (this is harmless if it occurs once at the start per stream)
frame=    2 fps=1.4 q=-0.0 size=       0kB time=00:00:00.06 bitrate=  46.9kbits/s speed=0.0466x     *** dropping frame 3 from stream 0 at ts 0
    Last message repeated 52 times
*** 100 dup!
frame=  105 fps= 49 q=-0.0 size=       2kB time=00:00:03.46 bitrate=   5.8kbits/s dup=99 drop=53 speed=1.61x     [out_0_0 @ 0x6541060] EOF on sink link out_0_0:default.
No more output streams to write to, finishing.
[h264 @ 0x64e8ab0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 121 >= 0
    Last message repeated 3 times
frame=  122 fps= 51 q=-0.0 Lsize=       3kB time=00:00:04.06 bitrate=   6.0kbits/s dup=99 drop=53 speed=1.72x   
video:3kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%
Input file #0 (test-320x240.h264):
  Input stream #0:0 (video): 105 packets read (641770 bytes); 76 frames decoded;
  Total: 105 packets (641770 bytes) demuxed
Output file #0 (test-output.h264):
  Output stream #0:0 (video): 122 frames encoded; 126 packets muxed (3038 bytes);
  Total: 126 packets (3038 bytes) muxed
76 frames successfully decoded, 0 decoding errors
[AVIOContext @ 0x652bcc0] Statistics: 0 seeks, 1 writeouts
[AVIOContext @ 0x616fcf0] Statistics: 641770 bytes read, 0 seeks
==18427==
==18427== HEAP SUMMARY:
==18427==     in use at exit: 655,210 bytes in 315 blocks
==18427==   total heap usage: 5,460 allocs, 5,145 frees, 4,251,344 bytes allocated
==18427==
==18427== 655,210 (2,520 direct, 652,690 indirect) bytes in 105 blocks are definitely lost in loss record 5 of 5
==18427==    at 0x48473FC: memalign (vg_replace_malloc.c:857)
==18427==    by 0x4847523: posix_memalign (vg_replace_malloc.c:1020)
==18427==    by 0x5EDF01F: av_malloc (mem.c:87)
==18427==    by 0x5EDF1EF: av_mallocz (mem.c:238)
==18427==    by 0x5ECB1E7: av_buffer_ref (buffer.c:95)
==18427==    by 0x4E01907: av_packet_ref (avpacket.c:618)
==18427==    by 0x4E78E0B: avcodec_send_packet (decode.c:662)
==18427==    by 0x42A447: decode (ffmpeg.c:2224)
==18427==    by 0x42A447: decode_video (ffmpeg.c:2368)
==18427==    by 0x42B213: process_input_packet (ffmpeg.c:2609)
==18427==    by 0x409EEB: process_input (ffmpeg.c:4445)
==18427==    by 0x409EEB: transcode_step (ffmpeg.c:4565)
==18427==    by 0x409EEB: transcode (ffmpeg.c:4619)
==18427==    by 0x409EEB: main (ffmpeg.c:4819)
==18427==
==18427== LEAK SUMMARY:
==18427==    definitely lost: 2,520 bytes in 105 blocks
==18427==    indirectly lost: 652,690 bytes in 210 blocks
==18427==      possibly lost: 0 bytes in 0 blocks
==18427==    still reachable: 0 bytes in 0 blocks
==18427==         suppressed: 0 bytes in 0 blocks
==18427==
==18427== For counts of detected and suppressed errors, rerun with: -v
==18427== Use --track-origins=yes to see where uninitialised values come from
==18427== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)

[Matrin Dørum]

I'm pretty sure avcodec is the correct component; it's avcodec/v4l2_* that's not freeing the packet, and it doesn't look like any of those files are using avdevice.

[Andriy Gelman]

Confirmed mem leak on ffmpeg 4.0, but the issue appears to be solved on git master.
Tested on Raspberry Pi 4 (when testing ffmpeg 4.0, I needed to add commit d61cf1b1ebc for handling YUV420P)

Can OP test git master on DART-SD410 device?