Avfoundation Audio stutters / sounds broken with screen capture

[kevin]

Summary of the bug:
I am trying to do a basic screen capture in OS X [10.10.3] with audio from
mic using avfoundation.
The recorded audio is stuttering, breaks, or there is short bursts.

How to reproduce:

ffmpeg -f avfoundation -i "1:0" -r 25  -c:v libx264 -preset fast -crf 22
-c:a libfdk_aac  -b:a 128k -ar 44100 -s 640x480 screen.mp4

ffmpeg version: git-master and also version N-71716-g7b94a2f
built on ... OS X 10.10.3

I tried different codec, bitrate, and audio is broken in all.
While the same exact command with -i "0:0" for FaceTime camera has great audio quality.

Here is link to the recorded file in which audio is broken:
play on web: ​https://www.dropbox.com/s/p7ej68mrhi2te8r/screen.mp4?dl=0
downloadable link: ​https://www.dropbox.com/s/p7ej68mrhi2te8r/screen.mp4?dl=1

 ffmpeg -f avfoundation -i "1:0" -r 25  -c:v libx264 -preset fast -crf 22
-c:a libfdk_aac  -b:a 128k -ar 44100 -s 640x480 screen.mp4
ffmpeg version 2.6.1 Copyright (c) 2000-2015 the FFmpeg developers
  built with Apple LLVM version 6.1.0 (clang-602.0.49) (based on LLVM
3.6.0svn)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/2.6.1 --enable-shared
--enable-pthreads --enable-gpl --enable-version3 --enable-hardcoded-tables
--enable-avresample --cc=clang --host-cflags= --host-ldflags=
--enable-libx264 --enable-libmp3lame --enable-libvo-aacenc --enable-libxvid
--enable-libfreetype --enable-libvorbis --enable-libvpx --enable-libfaac
--enable-libass --enable-ffplay --enable-libfdk-aac --enable-openssl
--enable-libopus --enable-libquvi --enable-libx265 --enable-nonfree
--enable-vda
  libavutil      54. 20.100 / 54. 20.100
  libavcodec     56. 26.100 / 56. 26.100
  libavformat    56. 25.101 / 56. 25.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 11.102 /  5. 11.102
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  1.100 /  1.  1.100
  libpostproc    53.  3.100 / 53.  3.100
[avfoundation @ 0x7f9b89001400] Selected pixel format (yuv420p) is not
supported by the input device.
[avfoundation @ 0x7f9b89001400] Supported pixel formats:
[avfoundation @ 0x7f9b89001400]   uyvy422
[avfoundation @ 0x7f9b89001400]   yuyv422
[avfoundation @ 0x7f9b89001400]   nv12
[avfoundation @ 0x7f9b89001400]   0rgb
[avfoundation @ 0x7f9b89001400]   bgr0
[avfoundation @ 0x7f9b89001400] Overriding selected pixel format to use
uyvy422 instead.
[avfoundation @ 0x7f9b89001400] Stream #0: not enough frames to estimate
rate; consider increasing probesize
Input #0, avfoundation, from '1:0':
  Duration: N/A, start: 0.368389, bitrate: N/A
    Stream #0:0: Video: rawvideo (UYVY / 0x59565955), uyvy422, 1440x900,
1000k tbr, 1000k tbn, 1000k tbc
    Stream #0:1: Audio: pcm_f32le, 44100 Hz, stereo, flt, 2822 kb/s
No pixel format specified, yuv422p for H.264 encoding chosen.
Use -pix_fmt yuv420p for compatibility with outdated media players.
[libx264 @ 0x7f9b89103200] using cpu capabilities: MMX2 SSE2Fast SSSE3
SSE4.1 Cache64
[libx264 @ 0x7f9b89103200] profile High 4:2:2, level 3.0, 4:2:2 8-bit
[libx264 @ 0x7f9b89103200] 264 - core 144 r2533 c8a773e - H.264/MPEG-4 AVC
codec - Copyleft 2003-2015 - http://www.videolan.org/x264.html - options:
cabac=1 ref=2 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=6 psy=1
psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1
cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=3
lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0
bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1
b_bias=0 direct=1 weightb=1 open_gop=0 weightp=1 keyint=250 keyint_min=25
scenecut=40 intra_refresh=0 rc_lookahead=30 rc=crf mbtree=1 crf=22.0
qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'screen.mp4':
  Metadata:
    encoder         : Lavf56.25.101
    Stream #0:0: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv422p,
640x480, q=-1--1, 25 fps, 12800 tbn, 25 tbc
    Metadata:
      encoder         : Lavc56.26.100 libx264
    Stream #0:1: Audio: aac (libfdk_aac) ([64][0][0][0] / 0x0040), 44100
Hz, stereo, s16, 128 kb/s
    Metadata:
      encoder         : Lavc56.26.100 libfdk_aac
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
  Stream #0:1 -> #0:1 (pcm_f32le (native) -> aac (libfdk_aac))
Press [q] to stop, [?] for help
frame=  255 fps= 25 q=-1.0 Lsize=     286kB time=00:00:10.19 bitrate=
229.8kbits/s dup=102 drop=0
video:137kB audio:136kB subtitle:0kB other streams:0kB global headers:0kB
muxing overhead: 4.737433%
[libx264 @ 0x7f9b89103200] frame I:2     Avg QP:20.52  size: 46726
[libx264 @ 0x7f9b89103200] frame P:65    Avg QP:23.32  size:   640
[libx264 @ 0x7f9b89103200] frame B:188   Avg QP:31.27  size:    22
[libx264 @ 0x7f9b89103200] consecutive B-frames:  1.6%  0.0%  1.2% 97.3%
[libx264 @ 0x7f9b89103200] mb I  I16..4: 15.7% 48.2% 36.1%
[libx264 @ 0x7f9b89103200] mb P  I16..4:  0.0%  0.0%  0.4%  P16..4:  0.8%
 0.1%  0.0%  0.0%  0.0%    skip:98.7%
[libx264 @ 0x7f9b89103200] mb B  I16..4:  0.0%  0.0%  0.0%  B16..8:  0.2%
 0.0%  0.0%  direct: 0.0%  skip:99.8%  L0:38.8% L1:61.2% BI: 0.0%
[libx264 @ 0x7f9b89103200] 8x8 transform intra:43.0% inter:2.5%
[libx264 @ 0x7f9b89103200] coded y,uvDC,uvAC intra: 37.2% 37.7% 33.9%
inter: 0.0% 0.1% 0.0%
[libx264 @ 0x7f9b89103200] i16 v,h,dc,p: 60% 40%  0%  0%
[libx264 @ 0x7f9b89103200] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 87%  8%  3%  0%
 0%  0%  0%  0%  1%
[libx264 @ 0x7f9b89103200] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 23% 39% 12%  3%
 3%  3%  6%  3%  8%
[libx264 @ 0x7f9b89103200] i8c dc,h,v,p: 74% 11% 14%  1%
[libx264 @ 0x7f9b89103200] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x7f9b89103200] ref P L0: 73.8% 26.2%
[libx264 @ 0x7f9b89103200] ref B L0: 95.1%  4.9%
[libx264 @ 0x7f9b89103200] ref B L1: 95.1%  4.9%
[libx264 @ 0x7f9b89103200] kb/s:109.18

[kevin]

The screen captured video file with broken audio

[kevin]

Mailing list discussion of this problem:

https://ffmpeg.org/pipermail/ffmpeg-user/2015-April/026160.html

[Thilo Borgmann]

There is a patch that might help with the audio issues while screen capturing that can be found in ticket #4437.

Please test that patch.

[maddanio]

I had the same problem and tested the patch and audio is fine now!

[kevin]

is the patch applied or integrated into the main ffmpeg version? thanks!

[Thilo Borgmann]

An alternative patch also fixing this issue is currently in review on ffmpeg-devel:

​http://thread.gmane.org/gmane.comp.video.ffmpeg.devel/203959/focus=203966

[fthiery]

Hello,

For information, our issue was fixed by the patch provided in #4437 but not by the larger overhaul that you mention.

[LordHDL]

How do I apply this 3 part patch? When I get this in the console:

"fatal: patch with only garbage at line 9"

[Thilo Borgmann]

Thanks for the feedback!

The overhaul patch is going to be void soon since we're currently trying to get a common codebase with Libav. This will unfortunately postpone a real update on this device for some time.

However, thanks to your feedback, I will reevaluate again the corresponding tickets for the avfoundation device.

Thanks!

@LordHDL: sounds like a copy&paste issue. Either retry to download the patches again or clone from: ​https://github.com/thiloborgmann/FFmpeg.git
And checkout branch avf2.

[Demian Rodriguez]

Can someone please confirm is this bug still exists? I facing the same problem, audio stutters while capturing screen and saving mp4 file using h264 and aac.

[Carl Eugen Hoyos]

Replying to demian85:

Can someone please confirm is this bug still exists?

Didn't you confirm yesterday that this bug is still reproducible?

I facing the same problem, audio stutters while capturing screen and saving mp4 file using h264 and aac.

Please test the patch attached to ticket #4437.

[nickcrabtree]

Confirmed still broken in the latest git HEAD.

Also confirmed that the patch https://ffmpeg.org/pipermail/ffmpeg-devel/2015-August/177845.html fixes the issue for me when applied to the commit it was as generated for, although I had to increase the hardcoded QUEUE_SIZE from 4 to 400.

However the two-year-old patch does not apply cleanly to HEAD.

[Carl Eugen Hoyos]

I thought I had tested the patch applies to current FFmpeg git head, could you elaborate?

[nickcrabtree]

Console log:

NJCMBP15:~/soft/FFmpeg.current nickc Thu Jan 25 17:42:28
$ git checkout HEAD
Your branch is up-to-date with 'origin/master'.
NJCMBP15:~/soft/FFmpeg.current nickc Thu Jan 25 17:43:37
$ git pull
Already up-to-date.
NJCMBP15:~/soft/FFmpeg.current nickc Thu Jan 25 17:43:44
$ git apply -v  ~/Downloads/avf.patch 
Checking patch libavdevice/avfoundation.m...
error: while searching for:
        CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0.1, YES);
    }

    lock_frames(ctx);

    ctx->video_stream_index = stream->index;

    avpriv_set_pts_info(stream, 64, 1, avf_time_base);

    image_buffer      = CMSampleBufferGetImageBuffer(ctx->current_frame);
    image_buffer_size = CVImageBufferGetEncodedSize(image_buffer);

    stream->codec->codec_id   = AV_CODEC_ID_RAWVIDEO;

error: patch failed: libavdevice/avfoundation.m:551
error: libavdevice/avfoundation.m: patch does not apply

NJCMBP15:~/soft/FFmpeg.current nickc Thu Jan 25 17:46:51
$ git remote show origin
* remote origin
  Fetch URL: https://github.com/FFmpeg/FFmpeg.git
  Push  URL: https://github.com/FFmpeg/FFmpeg.git
  HEAD branch: master
  Remote branches:
    master       tracked
    oldabi       tracked
    release/0.10 tracked
    release/0.11 tracked
    release/0.5  tracked
    release/0.6  tracked
    release/0.7  tracked
    release/0.8  tracked
    release/0.9  tracked
    release/1.0  tracked
    release/1.1  tracked
    release/1.2  tracked
    release/2.0  tracked
    release/2.1  tracked
    release/2.2  tracked
    release/2.3  tracked
    release/2.4  tracked
    release/2.5  tracked
    release/2.6  tracked
    release/2.7  tracked
    release/2.8  tracked
    release/3.0  tracked
    release/3.1  tracked
    release/3.2  tracked
    release/3.3  tracked
    release/3.4  tracked
  Local branch configured for 'git pull':
    master merges with remote master
  Local ref configured for 'git push':
    master pushes to master (up to date)

[nickcrabtree]

My apologies. git apply -v -3 ~/Downloads/avf.patch applies just fine.

[Carl Eugen Hoyos]

Does the patch still fix the issue?

Sorry, I had tested with patch which defaults to a less strict mode.

[Rob Lowe]

@nickcrabtreeis is correct, applying the avf.patch and a queue size of 400 resolves the issue. 4 seems too small and fails

Thanks @nickcrabtreeis!

[arcsine]

This issue still seems to persist, and the avf.patch file seems to resolve it (with a queue size of 400). Is there anyway this can make it into a release? I'd rather not support a custom build with this patch if possible.

[Kevin Mark]

Can confirm that applying the patch against current master (4b7166c9d5) and increasing QUEUE_SIZE to 400 fixes this issue for me.

[shurhander]

Hey guys, still having the issue with latest ffmpeg version. Is there any plan to fix it or to apply the patch in ffmpeg's codebase?

[LordHDL]

Another vote for adding the patch to the main release. I no longer have a compiling environment for what I need so I can't make use of it. Attempts to patch and compile afterward just fail.

[shurhander]

Would someone be willing to help me compile it with the patch? I'd be super grateful. I've been trying but didn't manage to do it because the patch doesn't apply cleanly anymore

[Kevin Mark]

Patch from mailing list applied on top of 78e1d7f42110aec8d4cd703a7939c64b5a191952

[Kevin Mark]

Increase QUEUE_SIZE to 400. Applied on top of previous patch.

[Kevin Mark]

I've attached the patch from the mailing list rebased on the current master (78e1d7f42110aec8d4cd703a7939c64b5a191952) and an additional patch which increases the QUEUE_SIZE to 400. I can confirm it successfully builds although I haven't tested it again since I last posted. I'll post the commit message from that second patch here as it is relevant for this discussion:

By increasing the QUEUE_SIZE the chance of frames dropping out of the buffer is significantly reduced. It's been noted in trac #4513 by nickcrabtree that the current size of 4 is too low for most (any?) applications. 400 is ultimately an arbitrary-but-tested number.

I've noticed that even with QUEUE_SIZE set to 400 I can still run into a situation where it's not high enough as evidence by the log output. However this is always in instances where my system is consistently encoding at a rate slower than frames are being pulled from AVFoundation so the buffer overrun is inevitable unless the duration left to encode is short. So it's hard to put the blame on this code when the problem can be alleviated by changing a runtime configuration elsewhere.

Given the above limitation it seems clear that increasing QUEUE_SIZE is not a magic fix but gives the encoder more wiggle room to temporarily lag behind frames pulled in from AVFoundation. If you do see "video queue is full, the oldest frame has been dropped" in your log then increasing QUEUE_SIZE further may work in some instances but fixing the underlying issue would require an increase in encoder speed or a frame-rate reduction on the AVFoundation end.

One last note is that while I use the term encoder here (as that's likely to be the slowest part of your pipeline) I am really referring to anything downstream of AVFoundation's input.

[shurhander]

Thanks a lot for the new patch!!

[Takaki Makino]

Is anybody working for merging these patches into the master? They are really important if we want to encode 4k video on MacOS.

[Carl Eugen Hoyos]

If you want the patch applied, please test it and send it to the FFmpeg development mailing list.

[Thiago A. Correa]

This is still broken on 4.2.2 and the patch from #4437 fixes the issue but doesn't apply on top of 4.2.2 anymore.

[arcsine]

I see the ticket closed as duplicate, but I don't see a reference to the duplicated issues.

[arcsine]

.

[chris]

Replying to Correa:

This is still broken on 4.2.2 and the patch from #4437 fixes the issue but doesn't apply on top of 4.2.2 anymore.

do you know what the last version where this patch applied and ffmpeg was able to compile? i think at least adding a version number where this patch is applicable so people running macos who would like to do screen recording using both the screen and mic at the same time without audio issues would be beneficial. i'd much prefer to use ffmpeg for this rather than having the latest and greatest if this patch can not be applied as this issue still lingers in the latest version of ffmpeg installed via brew.

and just putting a version number here or a branch that people can check to apply this patch and get some basic A/V screen recording going would make for a decent point of reference.