Cb and Cr positions on GEQ filter

[Bernardo]

Summary of the bug:
The "cb(X, Y)" and "cr(X, Y)" functions of GEQ filter look shrunk when applied to a larger plane, like luminance or alpha planes.

ffmpeg -i file.mp4 -vf "geq=lum_expr=cb(X\,Y):cb_expr=0:cr_expr=0" out.mp4

The correct output is achieved only using half of each dimension, i.e. "lum_expr=cb(X/2\,Y/2)".

Setting "format=yuva444p" before the GEQ filter seems to fix the problem for "alpha_expr", but not for "lum_expr".

I attached one frame with the original picture and the filtered one just so you can see the effect.

Full output:

$ ffmpeg -y -v 99 -i M.mp4 -vf "geq=lum_expr=cr(X\,Y)+cb(X\,Y):cb_expr=0:cr_expr=0" -t 0.5 out.mp4
ffmpeg version N-41713-g53efb2f Copyright (c) 2000-2013 the FFmpeg developers
  built on Apr 30 2013 16:23:43 with gcc 4.7 (Ubuntu/Linaro 4.7.2-2ubuntu1)
  configuration: --enable-gpl --enable-libass --enable-libfaac --enable-libfdk-aac --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libspeex --enable-librtmp --enable-libtheora --enable-libvorbis --enable-libvpx --enable-x11grab --enable-libx264 --enable-nonfree --enable-version3 --enable-libfreetype --enable-libvidstab
  libavutil      52. 27.101 / 52. 27.101
  libavcodec     55.  6.100 / 55.  6.100
  libavformat    55.  3.100 / 55.  3.100
  libavdevice    55.  0.100 / 55.  0.100
  libavfilter     3. 61.101 /  3. 61.101
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  3.100 / 52.  3.100
Splitting the commandline.
Reading option '-y' ... matched as option 'y' (overwrite output files) with argument '1'.
Reading option '-v' ... matched as option 'v' (set logging level) with argument '99'.
Reading option '-i' ... matched as input file with argument 'M.mp4'.
Reading option '-vf' ... matched as option 'vf' (set video filters) with argument 'geq=lum_expr=cr(X\,Y)+cb(X\,Y):cb_expr=0:cr_expr=0'.
Reading option '-t' ... matched as option 't' (record or transcode "duration" seconds of audio/video) with argument '0.5'.
Reading option 'out.mp4' ... matched as output file.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option y (overwrite output files) with argument 1.
Applying option v (set logging level) with argument 99.
Successfully parsed a group of options.
Parsing a group of options: input file M.mp4.
Successfully parsed a group of options.
Opening an input file: M.mp4.
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1ebdca0] Format mov,mp4,m4a,3gp,3g2,mj2 probed with size=2048 and score=100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1ebdca0] ISO: File Type Major Brand: isom
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1ebdca0] File position before avformat_find_stream_info() is 2537955
[h264 @ 0x1ebea60] no picture 
[aac @ 0x1ec01a0] skip whole frame, skip left: 0
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1ebdca0] All info found
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x1ebdca0] File position after avformat_find_stream_info() is 76059
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'M.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf55.3.100
  Duration: 00:00:10.03, start: 0.015238, bitrate: 2023 kb/s
    Stream #0:0(und), 2, 1/24000: Video: h264 (High) (avc1 / 0x31637661), yuv420p, 854x480, 1001/48000, 1892 kb/s, 23.98 fps, 23.98 tbr, 24k tbn, 47.95 tbc
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(und), 1, 1/44100: Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 127 kb/s
    Metadata:
      handler_name    : SoundHandler
Successfully opened the file.
Parsing a group of options: output file out.mp4.
Applying option vf (set video filters) with argument geq=lum_expr=cr(X\,Y)+cb(X\,Y):cb_expr=0:cr_expr=0.
Applying option t (record or transcode "duration" seconds of audio/video) with argument 0.5.
Successfully parsed a group of options.
Opening an output file: out.mp4.
Successfully opened the file.
[Parsed_geq_0 @ 0x1ecf0e0] Setting 'lum_expr' to value 'cr(X,Y)+cb(X,Y)'
[Parsed_geq_0 @ 0x1ecf0e0] Setting 'cb_expr' to value '0'
[Parsed_geq_0 @ 0x1ecf0e0] Setting 'cr_expr' to value '0'
[graph 0 input from stream 0:0 @ 0x1ecf9a0] Setting 'video_size' to value '854x480'
[graph 0 input from stream 0:0 @ 0x1ecf9a0] Setting 'pix_fmt' to value '0'
[graph 0 input from stream 0:0 @ 0x1ecf9a0] Setting 'time_base' to value '1/24000'
[graph 0 input from stream 0:0 @ 0x1ecf9a0] Setting 'pixel_aspect' to value '0/1'
[graph 0 input from stream 0:0 @ 0x1ecf9a0] Setting 'sws_param' to value 'flags=2'
[graph 0 input from stream 0:0 @ 0x1ecf9a0] Setting 'frame_rate' to value '24000/1001'
[graph 0 input from stream 0:0 @ 0x1ecf9a0] w:854 h:480 pixfmt:yuv420p tb:1/24000 fr:24000/1001 sar:0/1 sws_param:flags=2
[format @ 0x1ed2940] compat: called with args=[yuv420p|yuvj420p|yuv422p|yuv444p]
[format @ 0x1ed2940] Setting 'pix_fmts' to value 'yuv420p|yuvj420p|yuv422p|yuv444p'
[AVFilterGraph @ 0x1ebdb80] query_formats: 4 queried, 3 merged, 0 already done, 0 delayed
[graph 1 input from stream 0:1 @ 0x1ecf8e0] Setting 'time_base' to value '1/44100'
[graph 1 input from stream 0:1 @ 0x1ecf8e0] Setting 'sample_rate' to value '44100'
[graph 1 input from stream 0:1 @ 0x1ecf8e0] Setting 'sample_fmt' to value 'fltp'
[graph 1 input from stream 0:1 @ 0x1ecf8e0] Setting 'channel_layout' to value '0x3'
[graph 1 input from stream 0:1 @ 0x1ecf8e0] tb:1/44100 samplefmt:fltp samplerate:44100 chlayout:0x3
[audio format for output stream 0:1 @ 0x1ed2fa0] Setting 'sample_fmts' to value 's16'
[audio format for output stream 0:1 @ 0x1ed2fa0] Setting 'channel_layouts' to value '0x4|0x3|0x7|0x107|0x37|0x3f'
[audio format for output stream 0:1 @ 0x1ed2fa0] auto-inserting filter 'auto-inserted resampler 0' between the filter 'Parsed_anull_0' and the filter 'audio format for output stream 0:1'
[AVFilterGraph @ 0x1ecff60] query_formats: 4 queried, 9 merged, 3 already done, 0 delayed
[auto-inserted resampler 0 @ 0x1ed4ac0] ch:2 chl:stereo fmt:fltp r:44100Hz -> ch:2 chl:stereo fmt:s16 r:44100Hz
[libx264 @ 0x1ece3e0] using mv_range_thread = 40
[libx264 @ 0x1ece3e0] using cpu capabilities: MMX2 SSE2Fast SSSE3 Cache64 SlowShuffle
[libx264 @ 0x1ece3e0] profile High, level 3.0
[libx264 @ 0x1ece3e0] 264 - core 130 r60 c832fe9 - H.264/MPEG-4 AVC codec - Copyleft 2003-2013 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=23 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
[h264 @ 0x1ebea60] detected 4 logical cores
Output #0, mp4, to 'out.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf55.3.100
    Stream #0:0(und), 0, 1/24000: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv420p, 854x480, 1001/24000, q=-1--1, 24k tbn, 23.98 tbc
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(und), 0, 1/44100: Audio: aac (libfaac) ([64][0][0][0] / 0x0040), 44100 Hz, stereo, s16, 128 kb/s
    Metadata:
      handler_name    : SoundHandler
Stream mapping:
  Stream #0:0 -> #0:0 (h264 -> libx264)
  Stream #0:1 -> #0:1 (aac -> libfaac)
Press [q] to stop, [?] for help
[h264 @ 0x28e0080] no picture 
[h264 @ 0x2924da0] no picture 
[aac @ 0x1ec01a0] skip whole frame, skip left: 0
No more output streams to write to, finishing.=00:00:00.37 bitrate=   1.0kbits/s    
[libx264 @ 0x1ece3e0] frame=   0 QP=20.41 NAL=3 Slice:I Poc:0   I:1620 P:0    SKIP:0    size=9783 bytes
[libx264 @ 0x1ece3e0] frame=   1 QP=24.24 NAL=2 Slice:P Poc:2   I:954  P:322  SKIP:344  size=7025 bytes
[libx264 @ 0x1ece3e0] frame=   2 QP=31.27 NAL=2 Slice:P Poc:4   I:873  P:372  SKIP:375  size=6762 bytes
[libx264 @ 0x1ece3e0] frame=   3 QP=23.57 NAL=2 Slice:P Poc:6   I:436  P:376  SKIP:808  size=5379 bytes
[libx264 @ 0x1ece3e0] frame=   4 QP=25.30 NAL=2 Slice:P Poc:8   I:1083 P:364  SKIP:173  size=7295 bytes
[libx264 @ 0x1ece3e0] frame=   5 QP=24.63 NAL=2 Slice:P Poc:10  I:958  P:445  SKIP:217  size=6568 bytes
[libx264 @ 0x1ece3e0] frame=   6 QP=25.03 NAL=2 Slice:P Poc:12  I:491  P:348  SKIP:781  size=7379 bytes
[libx264 @ 0x1ece3e0] frame=   7 QP=24.38 NAL=2 Slice:P Poc:14  I:453  P:391  SKIP:776  size=6856 bytes
[libx264 @ 0x1ece3e0] frame=   8 QP=23.95 NAL=2 Slice:P Poc:16  I:426  P:366  SKIP:828  size=5757 bytes
[libx264 @ 0x1ece3e0] frame=   9 QP=22.10 NAL=2 Slice:P Poc:18  I:559  P:328  SKIP:733  size=6413 bytes
[libx264 @ 0x1ece3e0] frame=  10 QP=23.98 NAL=2 Slice:P Poc:20  I:490  P:437  SKIP:693  size=6328 bytes
[libx264 @ 0x1ece3e0] frame=  11 QP=33.72 NAL=2 Slice:P Poc:22  I:417  P:290  SKIP:913  size=3812 bytes
frame=   12 fps=7.0 q=-1.0 Lsize=      89kB time=00:00:00.51 bitrate=1424.4kbits/s    
video:78kB audio:9kB subtitle:0 global headers:0kB muxing overhead 1.724562%
[libx264 @ 0x1ece3e0] frame I:1     Avg QP:20.41  size:  9783
[libx264 @ 0x1ece3e0] frame P:11    Avg QP:25.65  size:  6325
[libx264 @ 0x1ece3e0] mb I  I16..4: 75.1%  9.9% 15.1%
[libx264 @ 0x1ece3e0] mb P  I16..4: 37.7%  1.4%  0.9%  P16..4: 10.5%  6.6%  5.6%  0.0%  0.0%    skip:37.3%
[libx264 @ 0x1ece3e0] 8x8 transform intra:4.7% inter:30.2%
[libx264 @ 0x1ece3e0] coded y,uvDC,uvAC intra: 5.0% 0.0% 0.0% inter: 13.3% 0.0% 0.0%
[libx264 @ 0x1ece3e0] i16 v,h,dc,p: 58% 42%  0%  0%
[libx264 @ 0x1ece3e0] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 21% 48% 16%  2%  2%  2%  3%  2%  4%
[libx264 @ 0x1ece3e0] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 39% 23% 21%  4%  2%  2%  3%  3%  3%
[libx264 @ 0x1ece3e0] i8c dc,h,v,p: 100%  0%  0%  0%
[libx264 @ 0x1ece3e0] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x1ece3e0] ref P L0: 52.9% 11.1% 21.5% 14.5%
[libx264 @ 0x1ece3e0] kb/s:1268.44
55 frames successfully decoded, 0 decoding errors
[AVIOContext @ 0x1ec3840] Statistics: 60 seeks, 79 writeouts
[AVIOContext @ 0x1ec62e0] Statistics: 260780 bytes read, 2 seeks

[Bernardo]

Also, no external libraries are needed to reproduce the behavior.

[Elon Musk]

Other formats are subsamples yuvs, and thus you can not expect that cb(X,Y) will work like that. This is clrearly explained in documentation.

If you still do not understand problem use format=yuva444p before and after geq.

[Bernardo]

Replying to richardpl:

Other formats are subsampled yuvs, and thus you can not expect that cb(X,Y) will work like that. This is clearly explained in documentation.

If you still do not understand problem use format=yuva444p before and after geq.

Yes, but I was having the same problem even when using "format=yuva444p".
Looks like it's a problem just in FFplay... I checked that FFmpeg really works fine

ffplay -i file.mp4 -vf "format=yuva444p, geq=lum_expr=cb(X\,Y)+cr(X\,Y):cb_expr=0:cr_expr=0"

Could you please check again with the above command?

[Elon Musk]

Add format also after geq, otherwise ffplay will insert whatever other subsampled format it finds nice.

Beside there is a reason why SW & SH variables can be used....

[Bernardo]

Replying to richardpl:

Add format also after geq, otherwise ffplay will insert whatever other subsampled format it finds nice.

Well, I find it very odd to have to convert the format twice... Even on FFmpeg it may be possible to have this issue when applying only the first format?

[Elon Musk]

Itsn't doing any converting more than it needs, its just making sure that right format is passed to geq filter.