Opened 13 months ago

Last modified 4 months ago

#10149 new defect

Unable to read most of HLS M3U8 from Arte TV

Reported by: Thomas ERNEST Owned by:
Priority: normal Component: avformat
Version: 4.4.3 Keywords: hls m3u8 fmp4 multivariante
Cc: Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description (last modified by Thomas ERNEST)

Hello,

I am trying to improve Kodi plugin Arte+7. The improvement aims at reading videos from new Arte TV API instead of "old" Arte HBB TV API. Old content is full mp4 file. New content is HTTP Live Streaming (HLS) packaged in m3u8 files. Some m3u8 files can be read, some cannot.

How to reproduce:

% FAILURE
% ./ffplay.exe -v 9 -loglevel 99 https://arte-cmafhls.akamaized.net/am/cmaf/089000/089100/089171-000-A/230120220655/089171-000-A_VF-STF_XQ.m3u8 2> ko.log
...
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000228e9e50a00] type:'sidx' parent:'root' sz: 32248 847 9223372036854775807
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000228e9e50a00] sidx reference_type 1 is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000228e9e50a00] If you want to help, upload a sample of this file to https://streams.videolan.org/upload/ and contact the ffmpeg-devel mailing list. (ffmpeg-devel@ffmpeg.org)
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000228e9e50a00] error reading header
[AVIOContext @ 00000228e9aae980] Statistics: 35459 bytes read, 0 seeks
...


% SUCCESS
% ./ffplay.exe -v 9 -loglevel 99 https://arte-cmafhls.akamaized.net/am/cmaf/105000/105600/105610-000-A/230111071325/105610-000-A_VOF-STF_XQ.m3u8 2> ok.log

I reproduced the erroneous and success cases with Kodi 19.5 Matrix and 20.0 Nexus. I reproduced the issue directly with ffplay also and attach the logs of ffplay command here.

EDIT : Both streams have WEBVTT SPRITES. But webvtt for successful playback is not found 404.
Failing one :

Successful one :

Link to webvtt related tickets :

Thank you in advance

Attachments (15)

artetv_hls_m3u8_ko.log (102.5 KB ) - added by Thomas ERNEST 13 months ago.
Logs of failing case
artetv_hls_m3u8_ok.log (575.3 KB ) - added by Thomas ERNEST 13 months ago.
Logs of a succesful case
failing_089171-000-A_VF-STF_XQ.m3u8 (2.3 KB ) - added by Thomas ERNEST 12 months ago.
failing m3u8 with hls and webvtt
successful_105610-000-A_VOF-STF_XQ.m3u8 (2.3 KB ) - added by Thomas ERNEST 12 months ago.
valid m3u8 with hls and webvtt
020528-000-A_de.m3u8 (171 bytes ) - added by Thomas ERNEST 6 months ago.
german audio stream m3u for program 20528
020528-000-A_en.m3u8 (168 bytes ) - added by Thomas ERNEST 6 months ago.
english audio stream m3u for program 20528
020528-000-A_fr.m3u8 (168 bytes ) - added by Thomas ERNEST 6 months ago.
french audio stream m3u for program 20528
020528-000-A_vid.m3u8 (160 bytes ) - added by Thomas ERNEST 6 months ago.
720p video stream m3u for program 20528
020528-000-A_main.m3u8 (599 bytes ) - added by Thomas ERNEST 6 months ago.
main parent m3u for program 20528
ffplay_homemade_multi_lang_m3u.log (657.8 KB ) - added by Thomas ERNEST 6 months ago.
ffplay logs playing 20528 m3u set - not seekable
020528-000-A_de.2.m3u8 (144 bytes ) - added by Thomas ERNEST 6 months ago.
seekable example - german audio stream
020528-000-A_en.2.m3u8 (141 bytes ) - added by Thomas ERNEST 6 months ago.
seekable example - english audio stream
020528-000-A_fr.2.m3u8 (141 bytes ) - added by Thomas ERNEST 6 months ago.
seekable example - french audio stream
020528-000-A_vid.2.m3u8 (176 bytes ) - added by Thomas ERNEST 6 months ago.
seekable example - video stream
020528-000-A_main - Copie.m3u8 (599 bytes ) - added by Thomas ERNEST 6 months ago.
seekable example - main content index

Download all attachments as: .zip

Change History (23)

by Thomas ERNEST, 13 months ago

Attachment: artetv_hls_m3u8_ko.log added

Logs of failing case

by Thomas ERNEST, 13 months ago

Attachment: artetv_hls_m3u8_ok.log added

Logs of a succesful case

comment:1 by Thomas ERNEST, 12 months ago

Component: undeterminedavformat
Description: modified (diff)
Keywords: webvtt added
Version: unspecified4.4.3

by Thomas ERNEST, 12 months ago

failing m3u8 with hls and webvtt

by Thomas ERNEST, 12 months ago

valid m3u8 with hls and webvtt

comment:2 by Thomas ERNEST, 12 months ago

Description: modified (diff)

comment:3 by Thomas ERNEST, 12 months ago

Keywords: fmp4 subtitles added; kodi arte removed

comment:4 by quinkblack, 12 months ago

sidx is the direct reason leading to the failure, but it's not the root cause.

  1. Media initialization section byterange includes sidx. There is no benefical in HLS's case, although DASH can use sidx explicitly. How to interpret the sidx is a question, HLS spec doesn't help. If the byterange excluding sidx, the issue should be gone.
#EXT-X-MAP:URI="089171-000-A_v360.mp4",BYTERANGE="33087@0"
  1. FFmpeg mp4 demux has no issue to handle the sidx. You can test it with
ffplay  'https://arte-cmafhls.akamaized.net/am/cmaf/089000/089100/089171-000-A/230120220655/medias/089171-000-A_v360.mp4'

The issue happens when playback fmp4 inside HLS. The log message "sidx reference_type 1 is not implemented" is misleading. It's some IO error leads to this error, the fmp4 doesn't use sidx reference_type 1.

I don't have much time to dig further now.

comment:5 by zebul666, 11 months ago

there is also that ticket related to webvtt https://trac.ffmpeg.org/ticket/10169

I am able to play such erroneous case by downloading separately video mp4 stream, audio mp4 stream, and webvtt (by following each m3u8) (removing any STYLE blocks in webvtt like said in #10169) and combine/merge them into a file; for local reading.

comment:6 by Thomas ERNEST, 6 months ago

Keywords: multivariante added; webvtt subtitles removed

Indeed the problem is not related to webvtt or subtitles. Thank you for your help quinkblack and zebul666.

The root cause is related to the multivariant playlist https://developer.apple.com/documentation/http-live-streaming/creating-a-multivariant-playlist or byterange addressing.
It is possible to build a set of new m3u to index streams without multivariant and byterange addressing so that FFMPEG can play it.

Nevertheless, there is still a problem with these new m3u : the content is not seekable. It is not possible to fast-forward, rewind or jump to a moment in video.

To reproduce:

  1. GET https://www.arte.tv/api/rproxy/emac/v4/fr/tv/pages/HOME/
  2. pick any programId from the response e.g. 020528-000-A
  3. GET https://api.arte.tv/api/player/v2/config/fr/020528-000-A
  4. Read URLs to m3u in data.attributes.streams.url

Notes : Each m3u has different combinaition audio and subtibles e.g. original version with french subtitles, german language without subtitles... m3u have the same multivariant video streams. Different audio stream and subtitles combinaison.

  1. Pick a video stream and every audios streams. Create a m3u to encapsulate each stream.
  2. Create a parent m3u to index and add metadata to stream m3u.

Notes : See m3u attached to the ticket as an example. m3u from step 5 and 6 follow the pattern in https://developer.apple.com/documentation/http-live-streaming/adding-alternate-media-to-a-playlist.

Last edited 6 months ago by Thomas ERNEST (previous) (diff)

by Thomas ERNEST, 6 months ago

Attachment: 020528-000-A_de.m3u8 added

german audio stream m3u for program 20528

by Thomas ERNEST, 6 months ago

Attachment: 020528-000-A_en.m3u8 added

english audio stream m3u for program 20528

by Thomas ERNEST, 6 months ago

Attachment: 020528-000-A_fr.m3u8 added

french audio stream m3u for program 20528

by Thomas ERNEST, 6 months ago

Attachment: 020528-000-A_vid.m3u8 added

720p video stream m3u for program 20528

by Thomas ERNEST, 6 months ago

Attachment: 020528-000-A_main.m3u8 added

main parent m3u for program 20528

by Thomas ERNEST, 6 months ago

ffplay logs playing 20528 m3u set - not seekable

comment:7 by Thomas ERNEST, 6 months ago

If I add in the m3u8 files #EXT-X-TARGETDURATION:6360 - 6360 being content duration in seconds - then I have the ability to seek the playlist with ffplay 6.0, but not with Kodi 20.1 (integrating ffplay 4.4.1 as far as I recall).

  • #EXT-X-TARGETDURATION:6360

Please find in attachements a set of m3u8 which I can navigate through.

I added other fields after a first successfull test as far as I remember. I guess they don't influence ffplay to seek or not the playlist :

  • #EXT-X-PLAYLIST-TYPE:VOD
  • #EXT-X-MEDIA-SEQUENCE:0
  • #EXT-X-ENDLIST

by Thomas ERNEST, 6 months ago

Attachment: 020528-000-A_de.2.m3u8 added

seekable example - german audio stream

by Thomas ERNEST, 6 months ago

Attachment: 020528-000-A_en.2.m3u8 added

seekable example - english audio stream

by Thomas ERNEST, 6 months ago

Attachment: 020528-000-A_fr.2.m3u8 added

seekable example - french audio stream

by Thomas ERNEST, 6 months ago

Attachment: 020528-000-A_vid.2.m3u8 added

seekable example - video stream

by Thomas ERNEST, 6 months ago

seekable example - main content index

comment:8 by pedrojvpedrosa, 4 months ago

I have been getting this error on some of my HLS streams. I noticed it only happens on streams that are longer than around 8000 seconds (or 2666 segments).

Initially I thought this could be an issue with unsigned 32-bit integer overflow of the byterange addresses, but it also happens on the lighter variants such as 432p which don't have addresses that overflow a 32bit unsigned int.

Last edited 4 months ago by pedrojvpedrosa (previous) (diff)
Note: See TracTickets for help on using tickets.