Get color information from source device

[Diederick Niehorster]

As discussed in ​https://trac.ffmpeg.org/ticket/8454, if possible the dshow filter should provide information about the color range, color space, primaries, trc and chroma location of the frames a device provides. Some DirectShow devices do provide this information. I open an issue here with a partial patch, as a starting point and for discussion, before i send it to the list. I need help with the following:

1. Check that my conversion from DXVA color information to FFmpeg's AVColorRange, AVColorPrimaries and AVColorTransferCharacteristic is correct.
2. Help to convert the DXVA color information to AVColorSpace
3. Advice on how to provide this information to FFmpeg/the user. As far as i understand, this is not normally provided by the format, but by the codec, which (I think) it outside the purview of dshow/a format.

Thanks!

[Balling]

No. DXVA_VideoPrimaries_EBU3213 is not AVCOL_PRI_EBU3213. It is PAL, AVCOL_PRI_BT470BG. AVCOL_PRI_EBU3213 has nothing to do with EBU rec. 3213, why it is called like this? Well, cause some people are not very clever, it should have stayed one of RCA/JEDEC P22 (super)group of phosphors. It was renamed here: ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20190809000128.15807-1-rzumer@tebako.net/ we do not have gamma 2.0 and 1.8 either.

We do not have DXVA_NominalRange_48_208 or the other even more crazy MFNominalRange_64_127. What Micro$oft was thinking about?
​https://docs.microsoft.com/en-us/windows/win32/api/mfobjects/ne-mfobjects-mfnominalrange

See this patch to see how to set it globally. ​https://github.com/FFmpeg/FFmpeg/commit/0539f15bbb86c5e81af56454c94e2eddefd34b7c FATE will also need to be fixed.

[Diederick Niehorster]

Thanks, i have updated the patch, just correcting what i already based on your comments. Annoyingly, it seems e.g. BT.2020 is exposed by mediafoundation, but not things accessible through dshow... Still better have this than nothing.
Is it possible to also figure out AVColorSpace from these fields? The documentation is more complete here: ​https://docs.microsoft.com/en-us/windows/win32/api/dxva2api/ns-dxva2api-dxva2_extendedformat, which may help. Exact same fields otherwise.

[Balling]

It should be

case DXVA_VideoPrimaries_EBU3213:
    return AVCOL_PRI_BT470BG;

As I said. But also see here: ​https://docs.microsoft.com/en-us/windows/win32/api/dxva2api/ne-dxva2api-dxva2_videoprimaries BT.470-2 System M; EBU 3212 (sic, non-existant 3212 spec, funny, should be 3213 and they managed to mix up the BT.470 M with B,G)

Also, where is PQ and HLG tranfers??

I do not think there are there. ​https://chromium.googlesource.com/chromium/src/+/070e970240f13d29be8930f9316f39ace711a116/ui/gfx/color_space_win.cc#126

[Diederick Niehorster]

preliminary patch

[Diederick Niehorster]

Seems i failed to attach the updated patch, sorry. I have attached a new one where I also convert chroma location (one unknown to me there, DV PAL, which is horizontally and vertically cosited: ​http://www.adamwilt.com/pix-sampling.html, ​https://docs.microsoft.com/en-us/windows/win32/api/dxva2api/ne-dxva2api-dxva2_videochromasubsampling).

I also added the "newer" color spaces/trcs. They are not in DirectShow, one needs to use mediafoundation for those. Stupid since DirectShow isn't going anywhere... I did note that the enum values provided by both DXVA_VideoTransferFunction and MFVideoTransferFunction (and also DXVA_VideoPrimaries and MFVideoPrimaries) match and that the bitfields in DXVA_ExtendedFormat are sufficiently large to contain the additional values in the MFVideoxxx enums. Interestingly, MS even presents example code that wholesale plops the MFVideo values into a DXVA struct: ​https://docs.microsoft.com/en-us/windows/win32/api/dxva2api/ns-dxva2api-dxva2_videodesc. So perhaps thats unofficially supported, and maybe some devices use it. I see it used in the wild in software at least: ​https://github.com/Nevcairiel/LAVFilters/blob/ddef56ae155d436f4301346408f4fdba755197d6/decoder/LAVVideo/Media.cpp#L752. So i have just included those values too, in the worst case they do nothing. Can you check what i have added?

But also see here: ​​https://docs.microsoft.com/en-us/windows/win32/api/dxva2api/ne-dxva2api-dxva2_videoprimaries BT.470-2 System M; EBU 3212 (sic, non-existant 3212 spec, funny, should be 3213)

Yeah indeed, ​https://docs.microsoft.com/en-us/windows/win32/api/dxva2api/ne-dxva2api-dxva2_videoprimaries seems to suggest it should be AVCOL_PRI_BT470M, not AVCOL_PRI_BT470BG. But lets assume the docs are wrong on that point too, besides what you pointed out?

I have also added determination of color space, using VideoTransferMatrix, and falling back on a guess if VideoTransferMatrix is not known, but VideoPrimaries is.

Lastly, the discovered info is now exposed through the stream's codecpars, so it can actually be used.

Thanks for the help!

[Balling]

"one unknown to me there, DV PAL, which is horizontally and vertically cosited"

But that is what is used HDR too in HEVC spec for BT.2100 and is used on all HDR Blu-rays. It is signalled like this all over our codebase. ​https://github.com/FFmpeg/FFmpeg/blob/bc70684e74a185d7b80c8b80bdedda659cb581b8/libavutil/pixfmt.h#L599 I am not sure how does it apply to P010.

But lets assume the docs are wrong on that point too, besides what you pointed out?

Yes, the reference on that ITU-T H.273, or ISO/IEC 23091-2:2019 spec that is freely available from iso.com: ​https://standards.iso.org/ittf/PubliclyAvailableStandards/index.html
Yes, looking into them they really mixed up those, CIE III.C is System M, LOL. Sigh. I will open pull request against Micro repo on github. (That mix up is also noted on wikipedia, ha.)

[Balling]

Here is a pull request. Ha. ​https://github.com/MicrosoftDocs/sdk-api/pull/792

[Balling]

So for DV it should also be AVCHROMA_LOC_TOPLEFT.

Also

case 4:     // MFVideoTransferMatrix_BT2020_10
 ret = AVCOL_SPC_BT2020_NCL;
 	        break;
case 5:     // MFVideoTransferMatrix_BT2020_12
 ret = AVCOL_SPC_BT2020_NCL;
 	        break;

If MFVideoTransFunc_2020_const is not applied, if it is, it is CL.

That ITU signals CL in matrix, while crazy Micro in Transfer. Well...

[Balling]

My pull was merged by Micro. ​https://github.com/MicrosoftDocs/sdk-api/pull/792#event-4826699179 hehe.

[Diederick Niehorster]

Ok, in dshow_color_primaries i now check if MFVideoTransFunc_2020_const is applied, and symmetrically, in dshow_color_trc i now check if MFVideoTransferMatrix_BT2020_10 or MFVideoTransferMatrix_BT2020_12 is applied. Also fixed the DV PAL chroma location. I have updated the attached patch. Does this look all correct to you then: ready to go? The final patch will be more involved, as i ran into devices that expose the same formats twice, just once with the extended info attached and once not. So format selection logic needs to change to make sure the one with extended info gets selected if available. I'll just send that along with this to the devel list directly.

Great the Microsoft docs got fixed so quick, good you did that!

[Diederick Niehorster]

new more complete patch

[Balling]

Yeah, looks nice. The only moment I dunno is MFVideoPrimaries_DCI_P3. Is it DCI-P3 or actually Display P3 (DCI-D65 uses D65)? Sigh. Not documented by Micro.

Also, fallback to primaries/trc and stuff already happens here: ​https://github.com/FFmpeg/FFmpeg/blob/bc70684e74a185d7b80c8b80bdedda659cb581b8/libavfilter/vf_colorspace.c#L74

Also, is P010BE signalisation is actually big endian or is that (as Carl noted) actually little endian and dshow is just swapped??

Also see my patch for P010LE and P010BE support in nut.c and raw.c! ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20210601040239.2690-1-val.zapod.vz@gmail.com/ I need somebody to create FATE hashes for me though, as it is very hard to do so, there is insane amount of hashes to update see ​https://patchwork.ffmpeg.org/check/33651/ :( I am not capable, I am afraid, though I did it in dpx patch. ​https://patchwork.ffmpeg.org/project/ffmpeg/patch/20210601191828.6957-1-val.zapod.vz@gmail.com/

[Diederick Niehorster]

Yeah, looks nice. The only moment I dunno is MFVideoPrimaries_DCI_P3. Is it DCI-P3 or actually Display P3 (DCI-D65 uses D65)? Sigh. Not documented by Micro.

Lets leave it as is, best guess i guess.

Also, fallback to primaries/trc and stuff already happens here: ​​https://github.com/FFmpeg/FFmpeg/blob/bc70684e74a185d7b80c8b80bdedda659cb581b8/libavfilter/vf_colorspace.c#L74

Best to keep it here anyway, many signals don't go through that filter?

Also, is P010BE signalisation is actually big endian or is that (as Carl noted) actually little endian and dshow is just swapped??

Thats a question for the other patch, right? I have no clue, hope the original submitter can chime in on that (e.g. by trying it out once patches are in).

I haven't worked with this FATE thing before. Do both my patch here and the other patch trigger changes in these hashes? Then i could do them both at once. If my patch here doesn't, i'd hope to stay away from it ;) Proof's in the pudding, i'll give it a try.

[Balling]

both my patch here and the other patch trigger changes in these hashes?

No, since they will have to have an actual dshow device for this.

[Diederick Niehorster]

Ok, I tried FATE, but got some make fate errors (linker trouble) that i prefer not to dig into now. I'll leave this for now then, on focus on dshow proper. Will deal with FATE if/when i get there.

[Diederick Niehorster]

I've got the patch submitted with send-mail (SMTP config, see ​https://gist.github.com/jasonkarns/4354421, worked in one go!). Forgot to acknowledge your great help, or link to this issue, will do that in the probable revision :). Should we just close this issue and move to the mailing list?