Opened 9 years ago
Last modified 3 months ago
#5514 open defect
Interlaced HEVC Steam not Decoded Properly
Reported by: | Jose Santiago | Owned by: | |
---|---|---|---|
Priority: | normal | Component: | avcodec |
Version: | git-master | Keywords: | hevc |
Cc: | dhelsley@haivision.com, ffmpeg@tmm1.net, pszemus, MasterQuestionable | Blocked By: | |
Blocking: | Reproduced by developer: | yes | |
Analyzed by developer: | no |
Description
Summary of the bug:
I have a 480i interlaced HEVC elementary stream. The HM reference decoder decodes the stream and generates 720x480 interlaced frames. FFMPEG/FFPLAY decodes the stream and 720x240 progressive frames and FFPLAY plays them at half the frame rate with the first/second field/frames bouncing up and down a line because these are actually 2 different fields of the same frame.
How to reproduce:
ffmpeg -i src13_interlaced.265 src13_interaced.yuv ffplay src13_interlaced.265 $ ffmpeg -i src13_interlaced.265 src13_interaced.yuv ffmpeg version git-2016-04-27-7bccbee-VF Copyright (c) 2000-2016 the FFmpeg developers built with gcc 4.4.7 (GCC) 20120313 (Red Hat 4.4.7-16) configuration: --extra-version=VF --prefix=/mnt/kermit/work/git/ort/build/linux64/stage/ffmpeg --ld=/usr/bin/cc --target-os=linux --arch=x86_64 --cpu=x86_64 --enable-pic --enable-static --enable-shared --enable-rpath --enable-avfilter --enable-pthreads --enable-zlib --enable-bzlib --enable-runtime-cpudetect --enable-hardcoded-tables --disable-stripping --disable-doc --disable-dxva2 --disable-vaapi --disable-vda --disable-vdpau --yasmexe=/mnt/kermit/work/git/ort/build/linux64/tools/yasm/bin/yasm --sdl-config=/mnt/kermit/work/git/ort/build/linux64/stage/sdl/lib/../bin/sdl-config --extra-cflags='-I/mnt/kermit/work/git/ort/build/linux64/stage/zlib/include -I/mnt/kermit/work/git/ort/build/linux64/stage/bzip2/include -I/mnt/kermit/work/git/ort/build/linux64/stage/lzma/include -I/mnt/kermit/work/git/ort/build/linux64/stage/sdl/include -DVFBUILD_DISABLE_INTMATH_OPTIMIZATIONS=1' --extra-ldflags='-L/mnt/kermit/work/git/ort/build/linux64/stage/zlib/lib -L/mnt/kermit/work/git/ort/build/linux64/stage/bzip2/lib -L/mnt/kermit/work/git/ort/build/linux64/stage/lzma/lib -L/mnt/kermit/work/git/ort/build/linux64/stage/sdl/lib' libavutil 55. 19.100 / 55. 19.100 libavcodec 57. 28.203 / 57. 28.203 libavformat 57. 29.200 / 57. 29.200 libavdevice 57. 0.101 / 57. 0.101 libavfilter 6. 39.202 / 6. 39.202 libswscale 4. 0.100 / 4. 0.100 libswresample 2. 0.101 / 2. 0.101 VFAVUtil_MetaData_RunUnitTests() Completed Successfully. Input #0, hevc, from 'src13_interlaced.265': Duration: N/A, bitrate: N/A Stream #0:0: Video: hevc (Main), yuv420p(tv), 720x240, 30 fps, 30 tbr, 1200k tbn, 30 tbc Output #0, rawvideo, to 'src13_interaced.yuv': Metadata: encoder : Lavf57.29.200 Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 720x240, q=2-31, 200 kb/s, 30 fps, 30 tbn, 30 tbc Metadata: encoder : Lavc57.28.203 rawvideo Stream mapping: Stream #0:0 -> #0:0 (hevc (native) -> rawvideo (native)) Press [q] to stop, [?] for help frame= 336 fps=0.0 q=-0.0 size= 85050kB time=00:00:11.20 bitrate=62208.0kbitsframe= 337 fps=263 q=-0.0 size= 85303kB time=00:00:11.23 bitrate=62208.0kbitsframe= 522 fps=337 q=-0.0 Lsize= 132131kB time=00:00:17.40 bitrate=62208.0kbits/s speed=11.2x video:132131kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%
Attachments (2)
Change History (30)
comment:1 by , 9 years ago
comment:2 by , 9 years ago
The files are too large to upload to the ticket. 2.5MB limit. I have provided the following files for external download:
HEVC Elementary Stream:
http://162.97.176.4/src13_interlaced.265
Does not Play correctly with: ffplay src13_interlaced.265
FFMPEG Decoder Output:
http://162.97.176.4/src13_ffmpeg.yuv
Does not Play correctly with: ffplay -f rawvideo -pix_fmt yuv420p -framerate 30 -s 720x480 -i src13_ffmpeg.yuv Better but half framerate and field bounce and incorrect resolution with: ffplay -f rawvideo -pix_fmt yuv420p -framerate 30 -s 720x240 -i src13_ffmpeg.yuv
HM Reference Decoder Output:
http://162.97.176.4/src13_hm.yuv
Plays correctly with: ffplay -f rawvideo -pix_fmt yuv420p -framerate 30 -s 720x480 -i src13_hm.yuv
follow-up: 6 comment:4 by , 9 years ago
There are differences. In #4141, according to the comments the stream in question has field_seq_flag=1 but pic_struct=3, which is improper syntax, and the reference decoder does the same thing as FFMPEG (output half-height frames).
With this particular elementary stream, field_seq_flag=1 and pic_struct=1 or 2 depending on which field it is, which is valid interlaced syntax. The reference decoder produces proper interlaced output frames with this test vector.
comment:5 by , 9 years ago
Keywords: | hevc added |
---|---|
Reproduced by developer: | set |
Status: | new → open |
by , 9 years ago
Attachment: | src13_interlaced_cut.265 added |
---|
follow-up: 7 comment:6 by , 4 years ago
Replying to dhelsley:
There are differences. In #4141, according to the comments the stream in question has field_seq_flag=1 but pic_struct=3, which is improper syntax, and the reference decoder does the same thing as FFMPEG (output half-height frames).
With this particular elementary stream, field_seq_flag=1 and pic_struct=1 or 2 depending on which field it is, which is valid interlaced syntax. The reference decoder produces proper interlaced output frames with this test vector.
Please can you look into this example https://yadi.sk/d/bQkXgt900uEdkg (DVB-T2 ts stream from Ostankino, USSR channel 58, Russia)? Is it the second as well? BTW, about reference decoder, is that this one? https://sourceforge.net/p/mjpeg/mailman/message/9927377/ or what?
follow-up: 8 comment:7 by , 4 years ago
Replying to Balling:
Replying to dhelsley:
There are differences. In #4141, according to the comments the stream in question has field_seq_flag=1 but pic_struct=3, which is improper syntax, and the reference decoder does the same thing as FFMPEG (output half-height frames).
With this particular elementary stream, field_seq_flag=1 and pic_struct=1 or 2 depending on which field it is, which is valid interlaced syntax. The reference decoder produces proper interlaced output frames with this test vector.
Please can you look into this example https://yadi.sk/d/bQkXgt900uEdkg (DVB-T2 ts stream from Ostankino, USSR channel 58, Russia)? Is it the second as well? BTW, about reference decoder, is that this one? https://sourceforge.net/p/mjpeg/mailman/message/9927377/ or what?
Wow, old bug.
Your stream has field_seq_flag=1 and pic_struct=10,11 which is similar to our case (pic_struct=1,2 is top/bottom field, pic_struct=10,11 is top with next bottom/bottom with previous top or Top Field First).
Reference software can be found here https://hevc.hhi.fraunhofer.de/
comment:8 by , 4 years ago
Replying to dhelsley:
pic_struct=10,11 is top with next bottom/bottom
I am afraid it is more difficult than that? If you play in Potplayer it will oscillate between i and ib (at least there are no other crazy types https://sourceforge.net/p/mjpeg/mailman/message/9927377/)
Fraunhofer, of course, LOL. He is everywhere, even VoLTE EVS (enhanced voice service) or VVC/H.266.
comment:9 by , 4 years ago
Well, it was partially implemented in 876dada0b58b5ecc80b2a25eb5c33974a71c8eb2 and
7c8b65f688ea75496e278b7c042f2eda746f3eac
So when you do ffprobe src13_interlaced_cut.265 it says
Input #0, hevc, from 'C:\Users\ZAQU\Downloads\src13_interlaced_cut.265': Duration: N/A, bitrate: N/A Stream #0:0: Video: hevc (Main), yuv420p(tv, top first), 720x240, 30 fps, 30 tbr, 1200k tbn, 30 t
But with https://yadi.sk/d/bQkXgt900uEdkg it is not like that, it says it is progressive, that is incorrect. I think I should try -filter:v idet. (P.S. Idet does not work even on src13_interlaced_cut.265)
Also after 876dada0b58b5ecc80b2a25eb5c33974a71c8eb2 it is showing itself in ffprobe -show_frames but even there there are not enough parameters as it is not just
interlaced_frame=1 top_field_first=1
comment:10 by , 4 years ago
Cc: | added |
---|
comment:11 by , 4 years ago
I have a simple patch to make ffprobe report these hevc files as interlaced. Not sure if it is completely correct though, as the spec was hard for me to parse. See page 326 of Rec. ITU-T H.265 v7 (11/2019)
diff --git a/libavcodec/hevc_parser.c b/libavcodec/hevc_parser.c index b444b99955..da9b9019e2 100644 --- a/libavcodec/hevc_parser.c +++ b/libavcodec/hevc_parser.c @@ -65,7 +65,24 @@ static int hevc_parse_slice_header(AVCodecParserContext *s, H2645NAL *nal, sh->first_slice_in_pic_flag = get_bits1(gb); s->picture_structure = sei->picture_timing.picture_struct; - s->field_order = sei->picture_timing.picture_struct; + switch (s->picture_structure) { + case 0: + s->field_order = AV_FIELD_PROGRESSIVE; + break; + case 1: + case 9: + case 11: + s->field_order = AV_FIELD_TT; + break; + case 2: + case 10: + case 12: + s->field_order = AV_FIELD_BB; + break; + default: + s->field_order = AV_FIELD_UNKNOWN; + break; + } if (IS_IRAP_NAL(nal)) { s->key_frame = 1;
comment:12 by , 4 years ago
This makes ffprobe report the correct height, but does not help with playback at all.
diff --git a/libavcodec/hevc_parser.c b/libavcodec/hevc_parser.c index b444b99955..d00093d592 100644 --- a/libavcodec/hevc_parser.c +++ b/libavcodec/hevc_parser.c @@ -105,6 +105,10 @@ static int hevc_parse_slice_header(AVCodecParserContext *s, H2645NAL *nal, den = ps->sps->vui.vui_time_scale; } + if (ps->sps->ptl.general_ptl.interlaced_source_flag) { + avctx->height = s->height * 2; + } + if (num != 0 && den != 0) av_reduce(&avctx->framerate.den, &avctx->framerate.num, num, den, 1 << 30);
comment:13 by , 4 years ago
I think that interlaced_source_flag is not intended for check here, better check field_seq_flag for 1 as per spec ("field_seq_flag equal to 1 indicates that the CVS conveys pictures that represent fields" and further) and that is how it is done in https://gitlab.com/mbunkus/mkvtoolnix/-/commit/47058a6da2672944aa4544f8ecb7b9a79ba3f752#0dd3dcd183ebb1497b9a9e3522e4d11a174384a2_282_288
That will not fix #4141 of course, but that file there is not spec compliant. It can further be fixed by checking for picture_struct and interlaced_source_flag.
Indeed as I read the spec (and as mkver reads it), the fields (not frames! ) can be progressive in all cases, not only for frame doubling and trippling.
comment:14 by , 4 years ago
Workaround, works for pic_struct=10,11:
ffplay "test" -vcodec hevc_cuvid -vf weave,hwupload_cuda,yadif_cuda=mode=0:parity=0:deint=0,hwdownload,format=nv12
You may want to -bsf:v hevc_mp4toannexb it first.
comment:15 by , 4 years ago
I don't know why there are two tickets about essentially the same issue. So I will bump this one too, with fresh sample of DVB-T2 HEVC broadcast https://www.mediafire.com/file/dm849hm4ykqyzvj Should stay alive for a long time, as I see few samples are not available anymore.
comment:16 by , 3 years ago
Cc: | added |
---|
comment:17 by , 3 months ago
I have submitted a patch to support interlaced decoding in HEVCDEC. It seems works on all of the streams i have access too atm. https://ffmpeg.org//pipermail/ffmpeg-devel/2024-October/335529.html
comment:20 by , 3 months ago
Here is a diff -Naur patch that applies for me:
diff -Naur ort-vendor-ffmpeg-orig/src/libavcodec/hevc/hevcdec.c ort-vendor-ffmpeg-patched/src/libavcodec/hevc/hevcdec.c --- ort-vendor-ffmpeg-orig/src/libavcodec/hevc/hevcdec.c 2024-10-30 09:34:17.373950807 -0500 +++ ort-vendor-ffmpeg-patched/src/libavcodec/hevc/hevcdec.c 2024-10-30 10:09:25.139087029 -0500 @@ -359,7 +359,18 @@ avctx->profile = sps->ptl.general_ptl.profile_idc; avctx->level = sps->ptl.general_ptl.level_idc; - ff_set_sar(avctx, sps->vui.common.sar); + // There are some streams in the wild that were encode field pitcures + // and set double height aspect ratio so that some players that do not + // support interlaced HEVC display the field pictures with double height. + // Since we are now combining the field pictures into a single interlaced + // frame, fix the sample aspect ratio to restore the correct shape for the + // reconstructed interlaced frames. + if (ff_hevc_sei_pict_struct_is_field_picture(s->sei.picture_timing.picture_struct) && + sps->vui.common.sar.num == 1 && sps->vui.common.sar.den == 2) { + ff_set_sar(avctx, (AVRational){1, 1}); + } else { + ff_set_sar(avctx, sps->vui.common.sar); + } if (sps->vui.common.video_signal_type_present_flag) avctx->color_range = sps->vui.common.video_full_range_flag ? AVCOL_RANGE_JPEG @@ -3844,6 +3855,7 @@ dst->rpl = ff_refstruct_ref(src->rpl); dst->nb_rpl_elems = src->nb_rpl_elems; + dst->sei_pic_struct = src->sei_pic_struct; dst->poc = src->poc; dst->ctb_count = src->ctb_count; dst->flags = src->flags; @@ -3874,6 +3886,8 @@ av_freep(&s->md5_ctx); av_freep(&s->h274db); + ff_hevc_output_frame_construction_ctx_unref(s); + ff_container_fifo_free(&s->output_fifo); for (int layer = 0; layer < FF_ARRAY_ELEMS(s->layers); layer++) { @@ -3930,6 +3944,11 @@ s->local_ctx[0].logctx = avctx; s->local_ctx[0].common_cabac_state = &s->cabac; + if (ff_hevc_output_frame_construction_ctx_alloc(s) != 0 || + !s->output_frame_construction_ctx) { + return AVERROR(ENOMEM); + } + s->output_fifo = ff_container_fifo_alloc_avframe(0); if (!s->output_fifo) return AVERROR(ENOMEM); @@ -3984,6 +4003,8 @@ } } + ff_hevc_output_frame_construction_ctx_replace(s, s0); + for (int i = 0; i < FF_ARRAY_ELEMS(s->ps.vps_list); i++) ff_refstruct_replace(&s->ps.vps_list[i], s0->ps.vps_list[i]); @@ -4047,6 +4068,7 @@ s->sei.common.content_light = s0->sei.common.content_light; s->sei.common.aom_film_grain = s0->sei.common.aom_film_grain; s->sei.tdrdi = s0->sei.tdrdi; + s->sei.picture_timing = s0->sei.picture_timing; return 0; } diff -Naur ort-vendor-ffmpeg-orig/src/libavcodec/hevc/hevcdec.h ort-vendor-ffmpeg-patched/src/libavcodec/hevc/hevcdec.h --- ort-vendor-ffmpeg-orig/src/libavcodec/hevc/hevcdec.h 2024-10-30 09:34:17.375950807 -0500 +++ ort-vendor-ffmpeg-patched/src/libavcodec/hevc/hevcdec.h 2024-10-30 10:09:25.144087008 -0500 @@ -375,6 +375,10 @@ int ctb_count; int poc; + // SEI Picture Timing Picture Structure Type. + // HEVC_SEI_PicStructType. + int sei_pic_struct; + const HEVCPPS *pps; ///< RefStruct reference RefPicListTab *rpl; ///< RefStruct reference int nb_rpl_elems; @@ -490,6 +494,8 @@ struct FFRefStructPool *rpl_tab_pool; } HEVCLayerContext; +struct HEVCOutputFrameConstructionContext; + typedef struct HEVCContext { const AVClass *c; // needed by private avoptions AVCodecContext *avctx; @@ -508,6 +514,9 @@ /** 1 if the independent slice segment header was successfully parsed */ uint8_t slice_initialized; + // Interlaced Frame Construction Context. + struct HEVCOutputFrameConstructionContext *output_frame_construction_ctx; ///< RefStruct reference + struct ContainerFifo *output_fifo; HEVCParamSets ps; @@ -677,6 +686,10 @@ return 0; } +int ff_hevc_output_frame_construction_ctx_alloc(HEVCContext *s); +void ff_hevc_output_frame_construction_ctx_replace(HEVCContext *dst, HEVCContext *src); +void ff_hevc_output_frame_construction_ctx_unref(HEVCContext *s); + /** * Find frames in the DPB that are ready for output and either write them to the * output FIFO or drop their output flag, depending on the value of discard. diff -Naur ort-vendor-ffmpeg-orig/src/libavcodec/hevc/refs.c ort-vendor-ffmpeg-patched/src/libavcodec/hevc/refs.c --- ort-vendor-ffmpeg-orig/src/libavcodec/hevc/refs.c 2024-10-11 14:08:54.987769044 -0500 +++ ort-vendor-ffmpeg-patched/src/libavcodec/hevc/refs.c 2024-10-30 10:22:08.033505443 -0500 @@ -20,9 +20,13 @@ * License along with FFmpeg; if not, write to the Free Software * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA */ - + +#include "libavutil/avassert.h" #include "libavutil/mem.h" +#include "libavutil/pixdesc.h" #include "libavutil/stereo3d.h" +#include "libavutil/thread.h" +#include "libavutil/timestamp.h" #include "container_fifo.h" #include "decode.h" @@ -31,6 +35,99 @@ #include "progressframe.h" #include "refstruct.h" +typedef struct HEVCOutputFrameConstructionContext { + // Thread Data Access/Synchronization. + AVMutex mutex; + + // Decoder Output Tracking. + uint64_t dpb_counter; + int dpb_poc; + uint64_t dpb_poc_ooorder_counter; + + // Collect the First Field. + int have_first_field; + int first_field_poc; + int first_field_sei_pic_struct; + AVFrame *first_field; + + uint64_t orphaned_field_pictures; + + // Reconstructed Interlaced Frames From Field Pictures for Output. + AVFrame *constructed_frame; + + // Output Frame Counter. + uint64_t output_counter; + int output_poc; + uint64_t output_poc_ooorder_counter; +} HEVCOutputFrameConstructionContext; + +static void hevc_output_frame_construction_ctx_free(FFRefStructOpaque opaque, void *obj) +{ + HEVCOutputFrameConstructionContext * ctx = (HEVCOutputFrameConstructionContext *)obj; + + if (!ctx) + return; + + av_frame_free(&ctx->first_field); + av_frame_free(&ctx->constructed_frame); + av_assert0(ff_mutex_destroy(&ctx->mutex) == 0); +} + +int ff_hevc_output_frame_construction_ctx_alloc(HEVCContext *s) +{ + if (s->output_frame_construction_ctx) { + av_log(s->avctx, AV_LOG_ERROR, + "s->output_frame_construction_ctx is already set.\n"); + return AVERROR_INVALIDDATA; + } + + s->output_frame_construction_ctx = + ff_refstruct_alloc_ext(sizeof(*(s->output_frame_construction_ctx)), + 0, NULL, hevc_output_frame_construction_ctx_free); + if (!s->output_frame_construction_ctx) + return AVERROR(ENOMEM); + + av_assert0(ff_mutex_init(&s->output_frame_construction_ctx->mutex, NULL) == 0); + + return 0; +} + +void ff_hevc_output_frame_construction_ctx_replace(HEVCContext *dst, HEVCContext *src) +{ + ff_refstruct_replace(&dst->output_frame_construction_ctx, + src->output_frame_construction_ctx); +} + +void ff_hevc_output_frame_construction_ctx_unref(HEVCContext *s) +{ + if (s->output_frame_construction_ctx && + ff_refstruct_exclusive(s->output_frame_construction_ctx)) { + + HEVCOutputFrameConstructionContext * ctx = s->output_frame_construction_ctx; + + ff_mutex_lock(&ctx->mutex); + + if (ctx->dpb_counter) { + av_log(s->avctx, AV_LOG_ERROR, + "[HEVCOutputFrameConstructionContext @ 0x%p]:\n" + " DPB: Counter=%" PRIu64 " POCOutOfOrder=%" PRIu64 " Orphaned=%" PRIu64 "\n" + " Output: Counter=%" PRIu64 " POCOutOfOrder=%" PRIu64 "\n" + "%s", + ctx, + ctx->dpb_counter, + ctx->dpb_poc_ooorder_counter, + ctx->orphaned_field_pictures, + ctx->output_counter, + ctx->output_poc_ooorder_counter, + ""); + } + + ff_mutex_unlock(&ctx->mutex); + } + + ff_refstruct_unref(&s->output_frame_construction_ctx); +} + void ff_hevc_unref_frame(HEVCFrame *frame, int flags) { frame->flags &= ~flags; @@ -151,11 +248,15 @@ for (j = 0; j < frame->ctb_count; j++) frame->rpl_tab[j] = frame->rpl; - if (s->sei.picture_timing.picture_struct == AV_PICTURE_STRUCTURE_TOP_FIELD) - frame->f->flags |= AV_FRAME_FLAG_TOP_FIELD_FIRST; - if ((s->sei.picture_timing.picture_struct == AV_PICTURE_STRUCTURE_TOP_FIELD) || - (s->sei.picture_timing.picture_struct == AV_PICTURE_STRUCTURE_BOTTOM_FIELD)) + frame->sei_pic_struct = s->sei.picture_timing.picture_struct; + if (ff_hevc_sei_pic_struct_is_interlaced(frame->sei_pic_struct)) { frame->f->flags |= AV_FRAME_FLAG_INTERLACED; + if (ff_hevc_sei_pic_struct_is_tff(frame->sei_pic_struct)) + frame->f->flags |= AV_FRAME_FLAG_TOP_FIELD_FIRST; + if (frame->sei_pic_struct == HEVC_SEI_PIC_STRUCT_FRAME_TFBFTF || + frame->sei_pic_struct == HEVC_SEI_PIC_STRUCT_FRAME_BFTFBF) + frame->f->repeat_pict = 1; + } ret = ff_hwaccel_frame_priv_alloc(s->avctx, &frame->hwaccel_picture_private); if (ret < 0) @@ -223,6 +324,81 @@ } } +static void copy_field2(AVFrame *_dst, const AVFrame *_src) +{ + const AVPixFmtDescriptor *desc = av_pix_fmt_desc_get(_src->format); + int i, j, planes_nb = 0; + for (i = 0; i < desc->nb_components; i++) + planes_nb = FFMAX(planes_nb, desc->comp[i].plane + 1); + for (i = 0; i < planes_nb; i++) { + int h = _src->height; + uint8_t *dst = _dst->data[i] + (_dst->linesize[i] / 2); + uint8_t *src = _src->data[i]; + if (i == 1 || i == 2) { + h = FF_CEIL_RSHIFT(_src->height, desc->log2_chroma_h); + } + for (j = 0; j < h; j++) { + memcpy(dst, src, _src->linesize[i]); + dst += _dst->linesize[i]; + src += _src->linesize[i]; + } + } +} + +static int interlaced_frame_from_fields(AVFrame *dst, + const AVFrame *field1, + const AVFrame *field2) +{ + int i, ret = 0; + + av_frame_unref(dst); + + dst->format = field1->format; + dst->width = field1->width; + dst->height = field1->height * 2; + dst->nb_samples = field1->nb_samples; + ret = av_channel_layout_copy(&dst->ch_layout, &field1->ch_layout); + if (ret < 0) + return ret; + + ret = av_frame_copy_props(dst, field1); + if (ret < 0) + return ret; + if (field1->duration > 0 && field1->duration != AV_NOPTS_VALUE) + dst->duration = field2->duration * 2; + else if (field2->duration > 0 && field2->duration != AV_NOPTS_VALUE) + dst->duration = field2->duration * 2; + + for (i = 0; i < field2->nb_side_data; i++) { + const AVFrameSideData *sd_src = field2->side_data[i]; + AVFrameSideData *sd_dst; + AVBufferRef *ref = av_buffer_ref(sd_src->buf); + sd_dst = av_frame_new_side_data_from_buf(dst, sd_src->type, ref); + if (!sd_dst) { + av_buffer_unref(&ref); + return AVERROR(ENOMEM); + } + } + + for (i = 0; i < AV_NUM_DATA_POINTERS; i++) + dst->linesize[i] = field1->linesize[i]*2; + + ret = av_frame_get_buffer(dst, 0); + if (ret < 0) + return ret; + + ret = av_frame_copy(dst, field1); + if (ret < 0) + av_frame_unref(dst); + + copy_field2(dst, field2); + + for (i = 0; i < AV_NUM_DATA_POINTERS; i++) + dst->linesize[i] = field1->linesize[i]; + + return ret; +} + int ff_hevc_output_frames(HEVCContext *s, unsigned layers_active_decode, unsigned layers_active_output, unsigned max_output, unsigned max_dpb, int discard) @@ -265,10 +441,232 @@ AVFrame *f = frame->needs_fg ? frame->frame_grain : frame->f; int output = !discard && (layers_active_output & (1 << min_layer)); + if (frame->poc != s->poc) { + if (s->avctx->active_thread_type == FF_THREAD_FRAME) + { + // Wait for other thread to finish decoding this frame/field picture. + // Otherwise I have seen image corruption for some streams.. + av_log(s->avctx, AV_LOG_DEBUG, + "Waiting on Frame POC: %d.\n", + frame->poc); + ff_progress_frame_await(&frame->tf, INT_MAX); + } + } else { + // This is the Context currently decoding.. + // Skip it to ensure that this frame is completely decoded and finalized. + // This will allow the next context to process it + // Otherwise I have seen image corruption for some streams. + av_log(s->avctx, AV_LOG_DEBUG, + "Schedule Frame for Next Pass POC: %d.\n", + frame->poc); + return 0; + } + + av_assert0(s->output_frame_construction_ctx); + av_assert0(ff_mutex_lock(&s->output_frame_construction_ctx->mutex) == 0); + if (output) { - f->pkt_dts = s->pkt_dts; - ret = ff_container_fifo_write(s->output_fifo, f); + const int dpb_poc = frame->poc; + const int dpb_sei_pic_struct = frame->sei_pic_struct; + AVFrame *output_frame = f; + int output_poc = dpb_poc; + int output_sei_pic_struct = dpb_sei_pic_struct; + + s->output_frame_construction_ctx->dpb_counter++; + if (s->output_frame_construction_ctx->dpb_counter > 1 && + dpb_poc < s->output_frame_construction_ctx->dpb_poc && + dpb_poc > 0) { + s->output_frame_construction_ctx->dpb_poc_ooorder_counter++; + av_log(s->avctx, AV_LOG_ERROR, + "DPB POC Out of Order POC %d < PrevPOC %d " + ": Counter=%" PRIu64 " OORCounter=%" PRIu64 ".\n", + dpb_poc, + s->output_frame_construction_ctx->dpb_poc, + s->output_frame_construction_ctx->dpb_counter, + s->output_frame_construction_ctx->dpb_poc_ooorder_counter); + } + s->output_frame_construction_ctx->dpb_poc = dpb_poc; + + if (ff_hevc_sei_pict_struct_is_field_picture(dpb_sei_pic_struct)) { + const int have_first_field = s->output_frame_construction_ctx->have_first_field; + const int is_first_field = + (ff_hevc_sei_pic_struct_is_tff(dpb_sei_pic_struct) && + ff_hevc_sei_pic_struct_is_tf(dpb_sei_pic_struct)) || + (ff_hevc_sei_pic_struct_is_bff(dpb_sei_pic_struct) && + ff_hevc_sei_pic_struct_is_bf(dpb_sei_pic_struct)) || + (!s->output_frame_construction_ctx->have_first_field && + (dpb_poc % 2) == 0) || + (s->output_frame_construction_ctx->have_first_field && + s->output_frame_construction_ctx->first_field_sei_pic_struct == dpb_sei_pic_struct && + (dpb_poc % 2) == 0 && + dpb_poc > s->output_frame_construction_ctx->first_field_poc); + + output_frame = NULL; + + if (!s->output_frame_construction_ctx->first_field) + { + s->output_frame_construction_ctx->first_field = av_frame_alloc(); + if (!s->output_frame_construction_ctx->first_field) { + av_log(s->avctx, AV_LOG_ERROR, "AVERROR(ENOMEM)"); + ret = AVERROR(ENOMEM); + goto unref_frame_and_check_ret; + } + } + if (!s->output_frame_construction_ctx->constructed_frame) { + s->output_frame_construction_ctx->constructed_frame = av_frame_alloc(); + if (!s->output_frame_construction_ctx->constructed_frame) { + av_log(s->avctx, AV_LOG_ERROR, "AVERROR(ENOMEM)"); + ret = AVERROR(ENOMEM); + goto unref_frame_and_check_ret; + } + } + + if (is_first_field) { + // This is a first field picture. + av_log(s->avctx, AV_LOG_DEBUG, + "Found first field picture POC %d.\n", + dpb_poc); + if (s->output_frame_construction_ctx->have_first_field) { + // We were waiting for a second field, but got another frist + // field instead. + av_log(s->avctx, AV_LOG_ERROR, + "Discarded Orphaned First Field with POC %d.\n", + s->output_frame_construction_ctx->first_field_poc); + } + s->output_frame_construction_ctx->have_first_field = 1; + s->output_frame_construction_ctx->first_field_sei_pic_struct = dpb_sei_pic_struct; + s->output_frame_construction_ctx->first_field_poc = dpb_poc; + ret = av_frame_ref(s->output_frame_construction_ctx->first_field, f); + if (ret < 0) { + av_log(s->avctx, AV_LOG_ERROR, + "Failure updating first Field picture POC %d.\n", + dpb_poc); + s->output_frame_construction_ctx->have_first_field = 0; + s->output_frame_construction_ctx->orphaned_field_pictures++; + goto unref_frame_and_check_ret; + } + } else if (have_first_field) { + // We Found the next field. + if (f->width == s->output_frame_construction_ctx->first_field->width && + f->height == s->output_frame_construction_ctx->first_field->height) { + // Combine the top and bottom fields into one frame for output. + AVFrame *constructed_frame = s->output_frame_construction_ctx->constructed_frame; + AVFrame *top_field; + AVFrame *bottom_field; + int tfPoc, bfPoc; + if (ff_hevc_sei_pic_struct_is_tf(dpb_sei_pic_struct)) { + top_field = f; + tfPoc = dpb_poc; + bottom_field = s->output_frame_construction_ctx->first_field; + bfPoc = s->output_frame_construction_ctx->first_field_poc; + } else { + top_field = s->output_frame_construction_ctx->first_field; + tfPoc = s->output_frame_construction_ctx->first_field_poc; + bottom_field = f; + bfPoc = dpb_poc; + } + ret = interlaced_frame_from_fields(constructed_frame, top_field, bottom_field); + if (ret >= 0) { + output_frame = constructed_frame; + output_poc = s->output_frame_construction_ctx->first_field_poc; + output_sei_pic_struct = s->output_frame_construction_ctx->first_field_sei_pic_struct; + output_frame->flags |= AV_FRAME_FLAG_INTERLACED; + if (!ff_hevc_sei_pic_struct_is_bf(output_sei_pic_struct)) { + output_frame->flags |= AV_FRAME_FLAG_TOP_FIELD_FIRST; + } else { + output_frame->flags &= ~AV_FRAME_FLAG_TOP_FIELD_FIRST; + } + } else { + av_log(s->avctx, AV_LOG_ERROR, + "Interlaced Frame Construction Failure POCs: %d %d.\n", + tfPoc, bfPoc); + s->output_frame_construction_ctx->orphaned_field_pictures += 2; + } + } else if ((dpb_poc % 2) == 0) { + av_log(s->avctx, AV_LOG_ERROR, + "Discarded orphaned first field pictures POC: %d.\n", + s->output_frame_construction_ctx->first_field_poc); + s->output_frame_construction_ctx->orphaned_field_pictures++; + // This may be the next first field. + s->output_frame_construction_ctx->have_first_field = 0; + av_assert0(ff_mutex_unlock(&s->output_frame_construction_ctx->mutex) == 0); + continue; + } else { + av_log(s->avctx, AV_LOG_ERROR, + "Discarded mismatched field pictures POCs: %d %d.\n", + s->output_frame_construction_ctx->first_field_poc, + dpb_poc); + s->output_frame_construction_ctx->orphaned_field_pictures++; + } + // Find the next first field. + s->output_frame_construction_ctx->have_first_field = 0; + } else { + // We have a second field without a first field. + av_log(s->avctx, AV_LOG_ERROR, + "Discarded orphaned second field picture with POC %d.\n", + dpb_poc); + s->output_frame_construction_ctx->orphaned_field_pictures++; + } + } else if (s->output_frame_construction_ctx->have_first_field) { + av_log(s->avctx, AV_LOG_ERROR, + "Discarded orphaned first field pictures POC: %d.\n", + s->output_frame_construction_ctx->first_field_poc); + s->output_frame_construction_ctx->orphaned_field_pictures++; + // Find the next first field. + s->output_frame_construction_ctx->have_first_field = 0; + } + + if (output_frame) { + output_frame->pkt_dts = s->pkt_dts; + + //av_log(s->avctx, AV_LOG_ERROR, + av_log(s->avctx, AV_LOG_DEBUG, + "s=0x%" PRIx64 " s->avctx=0x%" PRIx64 "\n" + " ====Output: FrameType:%s\n" + " === POC=%d PKTDTS=%s PTS=%s Duration=%s\n" + " === SEIPic=%d Interlaced=%s TFF=%s PictType='%c' Key=%s\n" + " === WxH=%dx%d SAR=%dx%d\n" + "%s", + (uint64_t)s, (uint64_t)s->avctx, + (output_frame->flags & AV_FRAME_FLAG_INTERLACED) ? "Interlaced" : "Progressive", + output_poc, + av_ts2str(output_frame->pkt_dts), + av_ts2str(output_frame->pts), + av_ts2str(output_frame->duration), + output_sei_pic_struct, + (output_frame->flags & AV_FRAME_FLAG_INTERLACED) ? "Yes" : "No", + (output_frame->flags & AV_FRAME_FLAG_TOP_FIELD_FIRST) ? "Yes" : "No", + av_get_picture_type_char(output_frame->pict_type), + (output_frame->flags & AV_FRAME_FLAG_KEY) ? "Yes" : "No", + output_frame->width, output_frame->height, + (int)output_frame->sample_aspect_ratio.num, + (int)output_frame->sample_aspect_ratio.den, + ""); + + s->output_frame_construction_ctx->output_counter++; + if (output_poc != dpb_poc && + s->output_frame_construction_ctx->output_counter > 1 && + output_poc < s->output_frame_construction_ctx->output_poc && + output_poc > 0) { + s->output_frame_construction_ctx->output_poc_ooorder_counter++; + av_log(s->avctx, AV_LOG_ERROR, + "Output POC Out of Order POC %d < PrevPOC %d " + ": Counter=%" PRIu64 " OORCounter=%" PRIu64 ".\n", + output_poc, + s->output_frame_construction_ctx->output_poc, + s->output_frame_construction_ctx->output_counter, + s->output_frame_construction_ctx->output_poc_ooorder_counter); + } + s->output_frame_construction_ctx->output_poc = output_poc; + + ret = ff_container_fifo_write(s->output_fifo, output_frame); + } } + +unref_frame_and_check_ret: + + av_assert0(ff_mutex_unlock(&s->output_frame_construction_ctx->mutex) == 0); + ff_hevc_unref_frame(frame, HEVC_FRAME_FLAG_OUTPUT); if (ret < 0) return ret; diff -Naur ort-vendor-ffmpeg-orig/src/libavcodec/hevc/sei.c ort-vendor-ffmpeg-patched/src/libavcodec/hevc/sei.c --- ort-vendor-ffmpeg-orig/src/libavcodec/hevc/sei.c 2024-10-11 14:08:54.988769042 -0500 +++ ort-vendor-ffmpeg-patched/src/libavcodec/hevc/sei.c 2024-10-30 10:09:25.151086977 -0500 @@ -67,21 +67,7 @@ return AVERROR_INVALIDDATA; if (sps->vui.frame_field_info_present_flag) { - int pic_struct = get_bits(gb, 4); - h->picture_struct = AV_PICTURE_STRUCTURE_UNKNOWN; - if (pic_struct == 2 || pic_struct == 10 || pic_struct == 12) { - av_log(logctx, AV_LOG_DEBUG, "BOTTOM Field\n"); - h->picture_struct = AV_PICTURE_STRUCTURE_BOTTOM_FIELD; - } else if (pic_struct == 1 || pic_struct == 9 || pic_struct == 11) { - av_log(logctx, AV_LOG_DEBUG, "TOP Field\n"); - h->picture_struct = AV_PICTURE_STRUCTURE_TOP_FIELD; - } else if (pic_struct == 7) { - av_log(logctx, AV_LOG_DEBUG, "Frame/Field Doubling\n"); - h->picture_struct = HEVC_SEI_PIC_STRUCT_FRAME_DOUBLING; - } else if (pic_struct == 8) { - av_log(logctx, AV_LOG_DEBUG, "Frame/Field Tripling\n"); - h->picture_struct = HEVC_SEI_PIC_STRUCT_FRAME_TRIPLING; - } + h->picture_struct = get_bits(gb, 4); } return 0; diff -Naur ort-vendor-ffmpeg-orig/src/libavcodec/hevc/sei.h ort-vendor-ffmpeg-patched/src/libavcodec/hevc/sei.h --- ort-vendor-ffmpeg-orig/src/libavcodec/hevc/sei.h 2024-10-11 14:08:54.988769042 -0500 +++ ort-vendor-ffmpeg-patched/src/libavcodec/hevc/sei.h 2024-10-30 10:15:30.231707140 -0500 @@ -38,10 +38,135 @@ typedef enum { - HEVC_SEI_PIC_STRUCT_FRAME_DOUBLING = 7, - HEVC_SEI_PIC_STRUCT_FRAME_TRIPLING = 8 + // SEI Picture Timing Picture Structure. + // From the ITU=T H.265 Standards Document v3 (04/2015): + // Table D.2: Interpretation of pic_struct: + // When present, pic_struct is constrained to use one of the following: + // - all pictures in CSV are one of: 0, 7 or 8. + // - all pictures in CSV are one of: 1, 2, 9, 10, 11, or 12.. + // - all pictures in CSV are one of: 3, 4, 5 or 6... + + // progressive frame. + HEVC_SEI_PIC_STRUCT_FRAME_PROGRESSIVE = 0, + + // top field. + HEVC_SEI_PIC_STRUCT_FIELD_TOP = 1, + // bottom field. + HEVC_SEI_PIC_STRUCT_FIELD_BOTTOM = 2, + + // top field, bottom field, in that order. Top Field First. + HEVC_SEI_PIC_STRUCT_FRAME_TFBF = 3, + // bottom Field, top field, in that order. Bottom Field First. + HEVC_SEI_PIC_STRUCT_FRAME_BFTF = 4, + + // top field, bottom field, top field repeated, Top Field First. + HEVC_SEI_PIC_STRUCT_FRAME_TFBFTF = 5, + // bottom field, top field, bottom field repeated, Bottom Field First. + HEVC_SEI_PIC_STRUCT_FRAME_BFTFBF = 6, + + // frame doubling. + HEVC_SEI_PIC_STRUCT_FRAME_DOUBLING = 7, + // frame trippling. + HEVC_SEI_PIC_STRUCT_FRAME_TRIPLING = 8, + + // top field paired with previous bottom field. Bottom Field First. + HEVC_SEI_PIC_STRUCT_FIELD_TFPBF = 9, + // bottom field paired with previous top field. Top Field First. + HEVC_SEI_PIC_STRUCT_FIELD_BFPTF = 10, + + // top field paired with next bottom field. Top Field First. + HEVC_SEI_PIC_STRUCT_FIELD_TFNBF = 11, + // bottom field paired with next top field. Bottom Field First. + HEVC_SEI_PIC_STRUCT_FIELD_BFNTF = 12, } HEVC_SEI_PicStructType; +// Returns 1 - when type is interlaced, 0 - otherwise. +static inline int ff_hevc_sei_pic_struct_is_interlaced(HEVC_SEI_PicStructType type) +{ + switch (type) { + case HEVC_SEI_PIC_STRUCT_FIELD_TOP: + case HEVC_SEI_PIC_STRUCT_FIELD_BOTTOM: + case HEVC_SEI_PIC_STRUCT_FRAME_TFBF: + case HEVC_SEI_PIC_STRUCT_FRAME_BFTF: + case HEVC_SEI_PIC_STRUCT_FRAME_TFBFTF: + case HEVC_SEI_PIC_STRUCT_FRAME_BFTFBF: + case HEVC_SEI_PIC_STRUCT_FIELD_TFPBF: + case HEVC_SEI_PIC_STRUCT_FIELD_BFPTF: + case HEVC_SEI_PIC_STRUCT_FIELD_TFNBF: + case HEVC_SEI_PIC_STRUCT_FIELD_BFNTF: + return 1; + default: + return 0; + } +} + +// Returns 1 - when type is top field first, 0 - otherwise. +static inline int ff_hevc_sei_pic_struct_is_tff(HEVC_SEI_PicStructType type) +{ + switch (type) { + case HEVC_SEI_PIC_STRUCT_FRAME_TFBF: + case HEVC_SEI_PIC_STRUCT_FRAME_TFBFTF: + case HEVC_SEI_PIC_STRUCT_FIELD_BFPTF: + case HEVC_SEI_PIC_STRUCT_FIELD_TFNBF: + return 1; + default: + return 0; + } +} + +// Returns 1 - when type is bottom field first, 0 - otherwise. +static inline int ff_hevc_sei_pic_struct_is_bff(HEVC_SEI_PicStructType type) +{ + switch (type) { + case HEVC_SEI_PIC_STRUCT_FRAME_BFTF: + case HEVC_SEI_PIC_STRUCT_FRAME_BFTFBF: + case HEVC_SEI_PIC_STRUCT_FIELD_TFPBF: + case HEVC_SEI_PIC_STRUCT_FIELD_BFNTF: + return 1; + default: + return 0; + } +} + +// Returns 1 - when type is top field, 0 - otherwise. +static inline int ff_hevc_sei_pic_struct_is_tf(HEVC_SEI_PicStructType type) +{ + switch (type) { + case HEVC_SEI_PIC_STRUCT_FIELD_TOP: + case HEVC_SEI_PIC_STRUCT_FIELD_TFPBF: + case HEVC_SEI_PIC_STRUCT_FIELD_TFNBF: + return 1; + default: + return 0; + } +} + +// Returns 1 - when type is bottom field, 0 - otherwise. +static inline int ff_hevc_sei_pic_struct_is_bf(HEVC_SEI_PicStructType type) +{ + switch (type) { + case HEVC_SEI_PIC_STRUCT_FIELD_BOTTOM: + case HEVC_SEI_PIC_STRUCT_FIELD_BFPTF: + case HEVC_SEI_PIC_STRUCT_FIELD_BFNTF: + return 1; + default: + return 0; + } +} + +// Returns 1 - when type is a field picture, 0 - otherwise. +static inline int ff_hevc_sei_pict_struct_is_field_picture(HEVC_SEI_PicStructType type) +{ + return (ff_hevc_sei_pic_struct_is_tf(type) || ff_hevc_sei_pic_struct_is_bf(type)) ? 1 : 0; +} + +// Returns 1 - when type is a frame picture, 0 - otherwise. +static inline int ff_hevc_sei_pict_struct_is_frame_picture(HEVC_SEI_PicStructType type) +{ + return ff_hevc_sei_pict_struct_is_field_picture(type) ? 0 : 1; +} + + typedef struct HEVCSEIPictureHash { uint8_t md5[3][16]; uint8_t is_md5;
comment:22 by , 3 months ago
Cc: | added |
---|
͏ Actually you may just push to:
͏ https://github.com/FFmpeg/FFmpeg
͏ ; and share the commit link for review...
͏ (seems everyone can make dangling commits there)
͏ See also: https://trac.ffmpeg.org/ticket/11194#comment:5
comment:23 by , 3 months ago
It literally says that merge requests are not allowed https://github.com/FFmpeg/FFmpeg/pulls
comment:24 by , 3 months ago
͏ Not creating PR.
͏ But push directly. (`git` CLI, or ͏"github.dev")
͏ It regardless serves as a good way for review.
͏ ----
͏ https://github.com/FFmpeg/FFmpeg/commit/861baf8bb785d2d70d926554116ca909fa74adc5
͏ https://github.com/FFmpeg/FFmpeg/compare/861baf8bb785d2d70d926554116ca909fa74adc5...1d4036ac85cff781b907eaef85acb38b09bc0c7c
͏ https://github.com/jlsantiago0/FFmpeg/commits/hevcdec-interlaced?since=2024-10-31
comment:25 by , 3 months ago
OK. I forked the GITHUB project and created a dev branch in my GITHUB repo:
https://github.com/jlsantiago0/FFmpeg/tree/hevcdec-interlaced
This branch has my patch.
comment:26 by , 3 months ago
I fixed a memory leak. The latest code can be found https://github.com/jlsantiago0/FFmpeg/tree/hevcdec-interlaced
Can you attach or upload the sample somewhere? Not much we can do otherwise