prores_ks encoder on Mac with M2 creates flickering

[jonnyburger]

Summary of the bug:

On an M2 Mac, encoding an image sequence with -profile:v 4 (4444) and the pixel format yuva444p10le creates a flickering video.

The issue was reproduced by two separated developers, but the bug does not happen on a M1 Mac. We suspect it has to do with the Apple Silicon Media Engine.

FFmpeg version: N-110004-g1231003c3c (build from latest development commit)

How to reproduce:

Get the reproduction from ​https://streams.videolan.org/ffmpeg/incoming/prores-repro.zip

Command

ffmpeg \
-r 30 \
-f image2 \
-s 1280x720 \
-start_number 0 \
-i out/element-%03d.png \
-vcodec prores_ks \
-profile:v 4 \
-pix_fmt yuva444p10le \
-map_metadata -1 \
-vendor apl0 \
-metadata comment="Made with Remotion 3.3.75" \
-y \
out.mov

[Thomas]

I'd like to add something: the problem is not when encoding on an M2 Mac on my end, but when decoding on an M2 Mac, which is a bigger problem since I can't solve this problem for M2 users when I encode a prores (with alpha) video on a Linux machine.

[Elon Musk]

So to clarify and to be able to reproduce, check which one/multiply are correct/true:

• decoding is not correct
• encoding is not correct
• decoding is not working properly on certain arch combo
• encoding is not working properly on certain arch combo
• when decoding always native ffmpeg decoder is used?

[Thomas]

Sorry about the misunderstanding.
Actually, this is the prores_ks encoding (4444 w/alpha) that is not correct, on all platforms.

Flickering only occurs when playing the video on a Mac M2 though, using native players such as Quicktime, iMovie, etc.

To reproduce:

• encode a prores 4444 w/alpha video on any platform
• play the video on a Mac M2 using Quicktime

[jonnyburger]

I agree with Thomas and the original bug report is unprecise.

The encoding is the same on both M1 and M2, but the playback on M2 is wrong.

To complete the reproduction: Take the reproduction from the first post, and play the video with QuickTime on macOS on M2.

To answer your question:

• decoding on macOS does not work correctly when the video is encoded with FFmpeg
• the QuickTime player is used for decoding
• It is not clear yet whether the fault is in the encoder or decoder
• The decoding works correctly on M1 arch, but not on M2 arch
• The encoding is presumably the same on both platforms, since it is software encoding.

My company remotion.dev is paying out a $500 bounty for the developer who can submit a patch!

This fix has value to us because it would allow users to export transparent videos and drag them into other video editing programs.

The bounty can be claimed by emailing ​jonny@remotion.dev.

[Kieran Kunhya]

Replying to Elon Musk:

So to clarify and to be able to reproduce, check which one/multiply are correct/true:

• decoding is not correct
• encoding is not correct
• decoding is not working properly on certain arch combo
• encoding is not working properly on certain arch combo
• when decoding always native ffmpeg decoder is used?

FFmpeg also has videotoolbox support for hardware decoding of ProRes so this should be checked too. Likely Quicktime is using this same decode API but I don't know for sure.

[jonnyburger]

I tried on a M2 Mac on a copy of FFmpeg (N-111888-gfc993e7a53), enabled with --enable-videotoolbox, as an additional step of my repro:

ffmpeg -hwaccel videotoolbox -i out.mov out.mp4 -y

It seems to have picked up, since values other than videotoolbox throw an error.

The video transcodes without any flickering, although quality is degraded.

I used ffmpeg to convert a video, because according of my knowledge, ffplay does not support the hwaccel flag.

[Clément Bœsch]

I'm looking into the issue currently.

[jonnyburger]

By the way, we (remotion.dev) have put out a $500 bounty to anyone can solve this: ​https://twitter.com/Remotion/status/1697141747426492422

[Clément Bœsch]

The following pending patch should address the issue: https://ffmpeg.org//pipermail/ffmpeg-devel/2023-December/318922.html

[jonnyburger]

For us, the patch solves the issue!

[Marth64]

Closing because patch was merged and user indicates it resolved the issue. Thank you.