Opened 9 years ago

Closed 9 years ago

#4749 closed defect (invalid)

firefox crashes in ffmpeg code (2.7.2 and git versions)

Reported by: zazdxscf Owned by:
Priority: important Component: avcodec
Version: git-master Keywords: crash aac
Cc: Michael Niedermayer Blocked By:
Blocking: Reproduced by developer: no
Analyzed by developer: no

Description

Summary of the bug:
How to reproduce:

I don't know how to reproduce, but having multiple tabs with at least one youtube video playing seems to trigger the segmentation fault in firefox, about twice per day.
I am currently trying (on latest git) 2 patches that I made from I can tell what's going on from the gdb stacktraces on the previous core dumps; that part with 'error: Cannot access memory at address'. But maybe I'm misinterpreting something so my patches are quite useless and it's only a matter of time before it crashes again (in which case I'll report back).

(i don't have the exact git commit number for the following paste, because I refreshed it to latest)

...
Core was generated by `firefox'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fdcef08d8cd in raise (sig=11) at ../sysdeps/unix/sysv/linux/pt-raise.c:36
36	  return INLINE_SYSCALL (tgkill, 3, pid, THREAD_GETMEM (THREAD_SELF, tid),
#0  0x00007fdcef08d8cd in raise (sig=11) at ../sysdeps/unix/sysv/linux/pt-raise.c:36
        resultvar = 0
        pid = <optimized out>
#1  0x00007fdcec0ac839 in nsProfileLock::FatalSignalHandler (signo=11, info=0x7fdc9b3fbab0, context=0x7fdc9b3fb980) at /usr/src/debug/www-client/firefox-39.0/mozilla-release/profile/dirserviceprovider/nsProfileLock.cpp:180
        unblock_sigs = {__val = {1024, 0 <repeats 15 times>}}
        oldact = <optimized out>
#2  <signal handler called>
No locals.
#3  0x00007fdcb235b34a in decode_spectrum_and_dequant (band_type=0x7fdcc5615d7c, ics=0x7fdcc5615100, pulse=0x7fdc9b3fbea0, pulse_present=0, sf=0x7fdcc561615c, gb=0x7fdc9b3fc2a0, coef=0x7fdcc56179c0, ac=0x7fdc864e3000) at /usr/src/debug/media-video/ffmpeg-9999/ffmpeg-9999/libavcodec/aacdec_template.c:1656
        code = <optimized out>
        nnz = <optimized out>
        cb_idx = <optimized out>
        bits = 0
        cf = 0x7fdcc5618060
        cb_vector_idx = 0x7fdcb28fe3a0 <codebook_vector02_idx>
        vlc_tab = 0x7fdcb2f62a80 <table>
        re_index = 2027
        re_cache = <optimized out>
        vq = 0x7fdcb28fe080 <codebook_vector10_vals>
        re_size_plus8 = 2056
        cbt_m1 = 2
        cfo = 0x7fdcc5618040
        off_len = 32
        group = <optimized out>
        g_len = 1
        i = 32
        k = <optimized out>
        g = 0
        idx = 32
        c = <optimized out>
        coef_base = 0x7fdcc56179c0
        offsets = 0x7fdcb28fdcc0 <swb_offset_1024_48>
#4  decode_ics (ac=ac@entry=0x7fdc864e3000, sce=sce@entry=0x7fdcc5615100, gb=gb@entry=0x7fdc9b3fc2a0, common_window=common_window@entry=1, scale_flag=0) at /usr/src/debug/media-video/ffmpeg-9999/ffmpeg-9999/libavcodec/aacdec_template.c:1958
        pulse = {num_pulse = 0, start = <optimized out>, pos = {<optimized out>, <optimized out>, <optimized out>, <optimized out>}, amp = {<optimized out>, <optimized out>, <optimized out>, <optimized out>}}
        tns = 0x7fdcc56151b0
        ics = 0x7fdcc5615100
        out = 0x7fdcc56179c0
        eld_syntax = <optimized out>
        er_syntax = <optimized out>
        pulse_present = 0
#5  0x00007fdcb235bd1c in decode_cpe (ac=ac@entry=0x7fdc864e3000, gb=gb@entry=0x7fdc9b3fc2a0, cpe=cpe@entry=0x7fdcc5607000) at /usr/src/debug/media-video/ffmpeg-9999/ffmpeg-9999/libavcodec/aacdec_template.c:2084
        i = <optimized out>
        ret = <optimized out>
        common_window = <optimized out>
        ms_present = 2
        eld_syntax = <optimized out>
#6  0x00007fdcb235cbd8 in aac_decode_frame_int (avctx=avctx@entry=0x7fdc8bbbd600, data=data@entry=0x7fdc9b3fc4f0, got_frame_ptr=got_frame_ptr@entry=0x7fdc9b3fc868, gb=gb@entry=0x7fdc9b3fc2a0, avpkt=avpkt@entry=0x7fdc9b3fc350) at /usr/src/debug/media-video/ffmpeg-9999/ffmpeg-9999/libavcodec/aacdec_template.c:2959
        ac = 0x7fdc864e3000
        che = 0x7fdcc5607000
        che_prev = <optimized out>
        elem_type_prev = TYPE_END
        err = 0
        elem_id = 0
        samples = 1024
        multiplier = <optimized out>
        audio_found = <optimized out>
        pce_found = <optimized out>
        is_dmono = <optimized out>
        sce_count = <optimized out>
#7  0x00007fdcb235dbea in aac_decode_frame (avctx=0x7fdc8bbbd600, data=0x7fdc9b3fc4f0, got_frame_ptr=0x7fdc9b3fc868, avpkt=0x7fdc9b3fc350) at /usr/src/debug/media-video/ffmpeg-9999/ffmpeg-9999/libavcodec/aacdec_template.c:3136
        ac = 0x7fdc864e3000
        buf = 0x7fdc67bfff00 "!\nT\365\266X\250\062\200\300\250\260\002fYiQK,T!&\327\360]Y\323[\337j\346\270⍛ \203\322­(*\204\212@\340-U@\371\324k\320\027\371\006\066z\333\332\372X\177\v\321EQ\356:U\362;\r\v\322\034\356\220:\275\016*y\267\201%\362\376\245\337\350\310\344su\360\205o\324'\227\347\234ݒ\v\234\t\006l\n\250"
        buf_size = 256
        gb = {buffer = 0x7fdc67bfff00 "!\nT\365\266X\250\062\200\300\250\260\002fYiQK,T!&\327\360]Y\323[\337j\346\270⍛ \203\322­(*\204\212@\340-U@\371\324k\320\027\371\006\066z\333\332\372X\177\v\321EQ\356:U\362;\r\v\322\034\356\220:\275\016*y\267\201%\362\376\245\337\350\310\344su\360\205o\324'\227\347\234ݒ\v\234\t\006l\n\250", buffer_end = 0x7fdc67c00000 <error: Cannot access memory at address 0x7fdc67c00000>, index = 2017, size_in_bits = 2048, size_in_bits_plus8 = 2056}
        buf_consumed = <optimized out>
        buf_offset = <optimized out>
        err = <optimized out>
        new_extradata_size = -1177434299
        jp_dualmono_size = 32732
        jp_dualmono = <optimized out>
#8  0x00007fdcb26ed1e1 in avcodec_decode_audio4 (avctx=0x7fdc8bbbd600, frame=frame@entry=0x7fdc9b3fc4f0, got_frame_ptr=got_frame_ptr@entry=0x7fdc9b3fc868, avpkt=avpkt@entry=0x7fdc9b3fc420) at /usr/src/debug/media-video/ffmpeg-9999/ffmpeg-9999/libavcodec/utils.c:2597
        side_size = 32732
        tmp = {buf = 0x0, pts = 0, dts = 0, data = 0x7fdc67bfff00 "!\nT\365\266X\250\062\200\300\250\260\002fYiQK,T!&\327\360]Y\323[\337j\346\270⍛ \203\322­(*\204\212@\340-U@\371\324k\320\027\371\006\066z\333\332\372X\177\v\321EQ\356:U\362;\r\v\322\034\356\220:\275\016*y\267\201%\362\376\245\337\350\310\344su\360\205o\324'\227\347\234ݒ\v\234\t\006l\n\250", size = 256, stream_index = 0, flags = 0, side_data = 0x0, side_data_elems = 0, duration = 0, destruct = 0x0, priv = 0x0, pos = 0, convergence_duration = 0}
        side = <optimized out>
        discard_padding = 0
        skip_reason = 0 '\000'
        discard_reason = 0 '\000'
        did_split = 0
        avci = 0x7fdcd33aefe0
        ret = 0

...

Here's a slightly different gdb backtrace which was done with an earlier(1 day?) latest git commit

...
Core was generated by `firefox'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f01579618cd in raise (sig=11) at ../sysdeps/unix/sysv/linux/pt-raise.c:36
36	  return INLINE_SYSCALL (tgkill, 3, pid, THREAD_GETMEM (THREAD_SELF, tid),
#0  0x00007f01579618cd in raise (sig=11) at ../sysdeps/unix/sysv/linux/pt-raise.c:36
        resultvar = 0
        pid = <optimized out>
#1  0x00007f01549ac839 in nsProfileLock::FatalSignalHandler (signo=11, info=0x7f00bc7fbab0, context=0x7f00bc7fb980) at /usr/src/debug/www-client/firefox-39.0/mozilla-release/profile/dirserviceprovider/nsProfileLock.cpp:180
        unblock_sigs = {__val = {1024, 0 <repeats 15 times>}}
        oldact = <optimized out>
#2  <signal handler called>
No locals.
#3  0x00007f011b72e46e in NEG_USR32 (s=<optimized out>, a=<optimized out>) at /usr/src/debug/media-video/ffmpeg-9999/ffmpeg-9999/libavcodec/x86/mathops.h:125
No locals.
#4  decode_spectrum_and_dequant (band_type=0x7f00f146cd7c, ics=0x7f00f146c100, pulse=0x7f00bc7fbea0, pulse_present=0, sf=0x7f00f146d15c, gb=0x7f00bc7fc2a0, coef=0x7f00f146e9c0, ac=0x7f00ddea7000) at /usr/src/debug/media-video/ffmpeg-9999/ffmpeg-9999/libavcodec/aacdec_template.c:1634
        n = <optimized out>
        nb_bits = <optimized out>
        index = <optimized out>
        code = <optimized out>
        cb_idx = <optimized out>
        cf = 0x7f00f146f360
        cb_vector_idx = 0x7f011bcd13a0 <codebook_vector02_idx>
        vlc_tab = 0x7f011c336760 <table>
        re_index = 2027
        re_cache = <optimized out>
        vq = 0x7f011bcd1448 <codebook_vector0_vals>
        re_size_plus8 = 2048
        cbt_m1 = 0
        cfo = 0x7f00f146f340
        off_len = 32
        group = <optimized out>
        g_len = 1
        i = 38
        k = <optimized out>
        g = 0
        idx = 38
        c = <optimized out>
        coef_base = 0x7f00f146e9c0
        offsets = 0x7f011bcd0cc0 <swb_offset_1024_48>
#5  decode_ics (ac=ac@entry=0x7f00ddea7000, sce=sce@entry=0x7f00f146c100, gb=gb@entry=0x7f00bc7fc2a0, common_window=common_window@entry=1, scale_flag=0) at /usr/src/debug/media-video/ffmpeg-9999/ffmpeg-9999/libavcodec/aacdec_template.c:1958
        pulse = {num_pulse = 0, start = <optimized out>, pos = {<optimized out>, <optimized out>, <optimized out>, <optimized out>}, amp = {<optimized out>, <optimized out>, <optimized out>, <optimized out>}}
        tns = 0x7f00f146c1b0
        ics = 0x7f00f146c100
        out = 0x7f00f146e9c0
        eld_syntax = <optimized out>
        er_syntax = <optimized out>
        pulse_present = 0
#6  0x00007f011b72ed1c in decode_cpe (ac=ac@entry=0x7f00ddea7000, gb=gb@entry=0x7f00bc7fc2a0, cpe=cpe@entry=0x7f00f145e000) at /usr/src/debug/media-video/ffmpeg-9999/ffmpeg-9999/libavcodec/aacdec_template.c:2084
        i = <optimized out>
        ret = <optimized out>
        common_window = <optimized out>
        ms_present = 2
        eld_syntax = <optimized out>
#7  0x00007f011b72fbd8 in aac_decode_frame_int (avctx=avctx@entry=0x7f0114708400, data=data@entry=0x7f00bc7fc4f0, got_frame_ptr=got_frame_ptr@entry=0x7f00bc7fc868, gb=gb@entry=0x7f00bc7fc2a0, avpkt=avpkt@entry=0x7f00bc7fc350) at /usr/src/debug/media-video/ffmpeg-9999/ffmpeg-9999/libavcodec/aacdec_template.c:2959
        ac = 0x7f00ddea7000
        che = 0x7f00f145e000
        che_prev = <optimized out>
        elem_type_prev = TYPE_END
        err = 0
        elem_id = 0
        samples = 1024
        multiplier = <optimized out>
        audio_found = <optimized out>
        pce_found = <optimized out>
        is_dmono = <optimized out>
        sce_count = <optimized out>
#8  0x00007f011b730bea in aac_decode_frame (avctx=0x7f0114708400, data=0x7f00bc7fc4f0, got_frame_ptr=0x7f00bc7fc868, avpkt=0x7f00bc7fc350) at /usr/src/debug/media-video/ffmpeg-9999/ffmpeg-9999/libavcodec/aacdec_template.c:3136
        ac = 0x7f00ddea7000
        buf = 0x7f00d8cfff00 "!\nU|-\a\001D\221P$\266\227.\267U\232i\330\004P\226\032"
        buf_size = 255
        gb = {buffer = 0x7f00d8cfff00 "!\nU|-\a\001D\221P$\266\227.\267U\232i\330\004P\226\032", buffer_end = 0x7f00d8cfffff "Z"<error: Cannot access memory at address 0x7f00d8d00000>, index = 2019, size_in_bits = 2040, size_in_bits_plus8 = 2048}
        buf_consumed = <optimized out>
        buf_offset = <optimized out>
        err = <optimized out>
        new_extradata_size = 556910405
        jp_dualmono_size = 32513
        jp_dualmono = <optimized out>
#9  0x00007f011bac01e1 in avcodec_decode_audio4 (avctx=0x7f0114708400, frame=frame@entry=0x7f00bc7fc4f0, got_frame_ptr=got_frame_ptr@entry=0x7f00bc7fc868, avpkt=avpkt@entry=0x7f00bc7fc420) at /usr/src/debug/media-video/ffmpeg-9999/ffmpeg-9999/libavcodec/utils.c:2597
        side_size = 32512
        tmp = {buf = 0x0, pts = 0, dts = 0, data = 0x7f00d8cfff00 "!\nU|-\a\001D\221P$\266\227.\267U\232i\330\004P\226\032", size = 255, stream_index = 0, flags = 0, side_data = 0x0, side_data_elems = 0, duration = 0, destruct = 0x0, priv = 0x0, pos = 0, convergence_duration = 0}
        side = <optimized out>
        discard_padding = 0
        skip_reason = 0 '\000'
        discard_reason = 0 '\000'
        did_split = 0
        avci = 0x7f011e449580
        ret = 0
#10 0x00007f011cbdf06d in gst_ffmpegauddec_audio_frame (ffmpegdec=ffmpegdec@entry=0x7f00f36552a0, data=data@entry=0x7f00d8cfff00 "!\nU|-\a\001D\221P$\266\227.\267U\232i\330\004P\226\032", size=<optimized out>, have_data=have_data@entry=0x7f00bc7fc868, outbuf=outbuf@entry=0x7f00bc7fc7d0, ret=ret@entry=0x7f00bc7fc86c, in_plugin=<optimized out>) at /usr/src/debug/media-plugins/gst-plugins-libav-1.4.5-r1/gst-libav-1.4.5/ext/libav/gstavauddec.c:475
        len = -1
        packet = {buf = 0x0, pts = 0, dts = 0, data = 0x7f00d8cfff00 "!\nU|-\a\001D\221P$\266\227.\267U\232i\330\004P\226\032", size = 255, stream_index = 0, flags = 0, side_data = 0x0, side_data_elems = 0, duration = 0, destruct = 0x0, priv = 0x0, pos = 0, convergence_duration = 0}
        frame = {data = {0x7f00e6855000 "\341\205G?e\035G?\224p\036?[R\350>\020o\330>l6\277>\264\372\276>\246\310\020?\337\374/?\231\024\061?\266\324K?\330\070W?kcJ?J\375[?\314\363W?\004\250)?\a\200\"?1\215/?\304\350)?ZD:?%H9?\262\345\004?\a}\352>\260\245\016?\v\023\365>w\371\237>\023\023\252>\237\332\363>\246o\360>\375hW>H\253\233\273\064a\b=\363\241\223=\024\267#=\314='>\256!\223>\215Q#>p\251\227\274\026\275\340\274\346\241\004=ǼS=\357^\303=\216\030Z>", 0x7f00f0e7e000 "\322\313!?Q\020\060?t#\a?[ڻ>\001ֿ>\035\213\274>\377Z\307>E^\017?\267\252\032?o\344\v?\270E2?\037\326T?\006<G?H\354H?\350{J?\347\335\062?_*4?\347&=?s{7?k\nD?\221(;?S", 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, linesize = {8192, 0, 0, 0, 0, 0, 0, 0}, extended_data = 0x7f00bc7fc4f0, width = 0, height = 0, nb_samples = 2048, format = 8, key_frame = 1, pict_type = AV_PICTURE_TYPE_NONE, base = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, sample_aspect_ratio = {num = 0, den = 1}, pts = -9223372036854775808, pkt_pts = 0, pkt_dts = -9223372036854775808, coded_picture_number = 0, display_picture_number = 0, quality = 0, reference = 0, qscale_table = 0x0, qstride = 0, qscale_type = 0, mbskip_table = 0x0, motion_val = {0x0, 0x0}, mb_type = 0x0, dct_coeff = 0x0, ref_index = {0x0, 0x0}, opaque = 0x0, error = {0, 0, 0, 0, 0, 0, 0, 0}, type = 1, repeat_pict = 0, interlaced_frame = 0, top_field_first = 0, palette_has_changed = 0, buffer_hints = 0, pan_scan = 0x0, reordered_opaque = -9223372036854775808, hwaccel_picture_private = 0x0, owner = 0x0, thread_opaque = 0x0, motion_subsample_log2 = 0 '\000', sample_rate = 44100, channel_layout = 3, buf = {0x7f00f92ff900, 0x7f00f92ff9c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, extended_buf = 0x0, nb_extended_buf = 0, side_data = 0x0, nb_side_data = 0, flags = 0, color_range = AVCOL_RANGE_UNSPECIFIED, color_primaries = AVCOL_PRI_RESERVED0, color_trc = AVCOL_TRC_RESERVED0, colorspace = AVCOL_SPC_RGB, chroma_location = AVCHROMA_LOC_UNSPECIFIED, best_effort_timestamp = -9223372036854775808, pkt_pos = 0, pkt_duration = 0, metadata = 0x0, decode_error_flags = 0, channels = 2, pkt_size = 255, qp_table_buf = 0x0}
...

or this for ffmpeg 2.7.2 version

...
Core was generated by `firefox'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f027fe598cd in raise (sig=11) at ../sysdeps/unix/sysv/linux/pt-raise.c:36
36	  return INLINE_SYSCALL (tgkill, 3, pid, THREAD_GETMEM (THREAD_SELF, tid),
#0  0x00007f027fe598cd in raise (sig=11) at ../sysdeps/unix/sysv/linux/pt-raise.c:36
        resultvar = 0
        pid = <optimized out>
#1  0x00007f027ceac839 in nsProfileLock::FatalSignalHandler (signo=11, info=0x7f021b1fbab0, context=0x7f021b1fb980) at /usr/src/debug/www-client/firefox-39.0/mozilla-release/profile/dirserviceprovider/nsProfileLock.cpp:180
        unblock_sigs = {__val = {1024, 0 <repeats 15 times>}}
        oldact = <optimized out>
#2  <signal handler called>
No locals.
#3  0x00007f024369f06e in NEG_USR32 (s=<optimized out>, a=<optimized out>) at /usr/src/debug/media-video/ffmpeg-2.7.2/ffmpeg-2.7.2/libavcodec/x86/mathops.h:125
No locals.
#4  decode_spectrum_and_dequant (band_type=0x7f0216c158f4, ics=0x7f0216c14c80, pulse=0x7f021b1fbea0, pulse_present=0, sf=0x7f0216c15cd4, gb=0x7f021b1fc2a0, coef=0x7f0216c17140, ac=0x7f021a2f9000) at /usr/src/debug/media-video/ffmpeg-2.7.2/ffmpeg-2.7.2/libavcodec/aacdec.c:1696
        n = <optimized out>
        nb_bits = <optimized out>
        index = <optimized out>
        code = <optimized out>
        cb_idx = <optimized out>
        cf = 0x7f0216c179b0
        cb_vector_idx = 0x7f0243c085c0 <codebook_vector02_idx>
        vlc_tab = 0x7f02442572c0 <table>
        re_index = 2024
        re_cache = <optimized out>
        vq = 0x7f0243c08668 <codebook_vector0_vals>
        re_size_plus8 = 2056
        cbt_m1 = 0
        cfo = 0x7f0216c17940
        off_len = 32
        group = <optimized out>
        g_len = 1
        i = 35
        k = <optimized out>
        g = 0
        idx = 35
        c = <optimized out>
        coef_base = 0x7f0216c17140
        offsets = 0x7f0243c07ee0 <swb_offset_1024_48>
#5  decode_ics (ac=ac@entry=0x7f021a2f9000, sce=sce@entry=0x7f0216c14c80, gb=gb@entry=0x7f021b1fc2a0, common_window=common_window@entry=1, scale_flag=0) at /usr/src/debug/media-video/ffmpeg-2.7.2/ffmpeg-2.7.2/libavcodec/aacdec.c:2010
        pulse = {num_pulse = 0, start = <optimized out>, pos = {<optimized out>, <optimized out>, <optimized out>, <optimized out>}, amp = {<optimized out>, <optimized out>, <optimized out>, <optimized out>}}
        tns = 0x7f0216c14d28
        ics = 0x7f0216c14c80
        out = 0x7f0216c17140
        eld_syntax = <optimized out>
        er_syntax = <optimized out>
        pulse_present = 0
#6  0x00007f024369f91c in decode_cpe (ac=ac@entry=0x7f021a2f9000, gb=gb@entry=0x7f021b1fc2a0, cpe=cpe@entry=0x7f0216c07000) at /usr/src/debug/media-video/ffmpeg-2.7.2/ffmpeg-2.7.2/libavcodec/aacdec.c:2121
        i = <optimized out>
        ret = <optimized out>
        common_window = <optimized out>
        ms_present = 1
        eld_syntax = <optimized out>
#7  0x00007f02436a07d9 in aac_decode_frame_int (avctx=avctx@entry=0x7f021c3c8600, data=data@entry=0x7f021b1fc4f0, got_frame_ptr=got_frame_ptr@entry=0x7f021b1fc868, gb=gb@entry=0x7f021b1fc2a0, avpkt=avpkt@entry=0x7f021b1fc350) at /usr/src/debug/media-video/ffmpeg-2.7.2/ffmpeg-2.7.2/libavcodec/aacdec.c:3015
        ac = 0x7f021a2f9000
        che = 0x7f0216c07000
        che_prev = <optimized out>
        elem_type_prev = TYPE_END
        err = 0
        elem_id = 0
        samples = 1024
        multiplier = <optimized out>
        audio_found = <optimized out>
        pce_found = <optimized out>
        is_dmono = <optimized out>
        sce_count = <optimized out>
#8  0x00007f02436a17ea in aac_decode_frame (avctx=0x7f021c3c8600, data=0x7f021b1fc4f0, got_frame_ptr=0x7f021b1fc868, avpkt=0x7f021b1fc350) at /usr/src/debug/media-video/ffmpeg-2.7.2/ffmpeg-2.7.2/libavcodec/aacdec.c:3192
        ac = 0x7f021a2f9000
        buf = 0x7f021a4fff00 "!\nO\"\356\377\377\352\222\317\ae(\305@F\343\064\211b\261\306U\302\302h4\027?\v\023\336\270#\004u\371\247?ضm\217́GM\203\f\301\034@\370\376\373\224\202o#l6+N\311A\\\233\225&d\266\376\334ir\276\367\bd\037\275BG\315Ɉ\t\v\276Y\317\064\022\202\240\321t\245QU\031\265\247\323*\273\232\067s+\235F\243M\374\343\370\025 \324\244R\b\003*D\016J\234\v\350\t\200HTf\n\001\nJC%\020\241\025\201\004ʶK\215\313\n\325E\240<\226\022~O\300\314l\341%_"
        buf_size = 256
        gb = {buffer = 0x7f021a4fff00 "!\nO\"\356\377\377\352\222\317\ae(\305@F\343\064\211b\261\306U\302\302h4\027?\v\023\336\270#\004u\371\247?ضm\217́GM\203\f\301\034@\370\376\373\224\202o#l6+N\311A\\\233\225&d\266\376\334ir\276\367\bd\037\275BG\315Ɉ\t\v\276Y\317\064\022\202\240\321t\245QU\031\265\247\323*\273\232\067s+\235F\243M\374\343\370\025 \324\244R\b\003*D\016J\234\v\350\t\200HTf\n\001\nJC%\020\241\025\201\004ʶK\215\313\n\325E\240<\226\022~O\300\314l\341%_", buffer_end = 0x7f021a500000 <error: Cannot access memory at address 0x7f021a500000>, index = 2009, size_in_bits = 2048, size_in_bits_plus8 = 2056}
        buf_consumed = <optimized out>
        buf_offset = <optimized out>
        err = <optimized out>
        new_extradata_size = 1278326597
        jp_dualmono_size = 32514
        jp_dualmono = <optimized out>
#9  0x00007f0243a0c041 in avcodec_decode_audio4 (avctx=0x7f021c3c8600, frame=frame@entry=0x7f021b1fc4f0, got_frame_ptr=got_frame_ptr@entry=0x7f021b1fc868, avpkt=avpkt@entry=0x7f021b1fc420) at /usr/src/debug/media-video/ffmpeg-2.7.2/ffmpeg-2.7.2/libavcodec/utils.c:2543
        side_size = 32514
        tmp = {buf = 0x0, pts = 0, dts = 0, data = 0x7f021a4fff00 "!\nO\"\356\377\377\352\222\317\ae(\305@F\343\064\211b\261\306U\302\302h4\027?\v\023\336\270#\004u\371\247?ضm\217́GM\203\f\301\034@\370\376\373\224\202o#l6+N\311A\\\233\225&d\266\376\334ir\276\367\bd\037\275BG\315Ɉ\t\v\276Y\317\064\022\202\240\321t\245QU\031\265\247\323*\273\232\067s+\235F\243M\374\343\370\025 \324\244R\b\003*D\016J\234\v\350\t\200HTf\n\001\nJC%\020\241\025\201\004ʶK\215\313\n\325E\240<\226\022~O\300\314l\341%_", size = 256, stream_index = 0, flags = 0, side_data = 0x0, side_data_elems = 0, duration = 0, destruct = 0x0, priv = 0x0, pos = 0, convergence_duration = 0}
        side = <optimized out>
        discard_padding = 0
        skip_reason = 0 '\000'
        discard_reason = 0 '\000'
        did_split = 0
        avci = 0x7f021a3deae0
        ret = 0
...

Ok now I see this:
"Patches should be submitted to the ffmpeg-devel mailing list and not this bug tracker."
but the rules there seem to be kinda strict https://ffmpeg.org/contact.html#MailingLists
I'll just drop the patches here for now, as they are.

Attachments (3)

handlebit_size0.patch​ (519 bytes ) - added by zazdxscf 9 years ago.
bit_size == 0 was not handled
use_init8.patch​ (530 bytes ) - added by zazdxscf 9 years ago.
use as init_get_bits8 in one place
backtrace_simple5.log​ (22.2 KB ) - added by zazdxscf 9 years ago.
gdb bt full with commit 5bf8590d6e and the 2 patches on top of it

Download all attachments as: .zip

Change History (12)

by zazdxscf, 9 years ago

Attachment: handlebit_size0.patch​ added

bit_size == 0 was not handled

by zazdxscf, 9 years ago

Attachment: use_init8.patch​ added

use as init_get_bits8 in one place

comment:1 by Carl Eugen Hoyos, 9 years ago

Component: ffmpeg → avcodec
Keywords: crash aac added; init_get_bits init_get_bits8 bit_size removed
Priority: normal → important

Please send your patches - made with git format-patch - to the development mailing list, patches are usually ignored here.

comment:2 by zazdxscf, 9 years ago

I'll give it a couple of days to see if it crashes again (because without the patch(es) it crashed like 4-5 times in the past 2 days). If it doesn't, then I'll proceed as instructed. Thanks.

comment:3 by Michael Niedermayer, 9 years ago

use_init8.patch, applied

Is there evidence that bit_size == 0 is actually happening ?

about the crashes, ive looked in the calling code in gstavauddec.c, and that is broken in the version i found, The size of a AVFrame is not part of the ABI thus sizeof(AVFrame) is not safe nor is creating it on the stack like its done.
This of course may be unrelated ...

The other potential cause of this crash would be lack of FF_INPUT_BUFFER_PADDING_SIZE bytes extra allocation for the input to avcodec_decode_audio4(), iam not sure if this is missing or not. But i see code dealing with that padding in gstavviddec.c but not gstavauddec.c

comment:4 by Michael Niedermayer, 9 years ago

Cc: Michael Niedermayer added

comment:5 by zazdxscf, 9 years ago

Thanks for applying that patch. (saved me some trouble figuring out how to get it to the mail list later on)

No evidence for that (bit_size == 0) as far as I know, but I just wanted to be extra-sure just in case, without thinking too much about it.

I haven't experienced anymore crashes yet, but I am keeping all software as is(not updating anything on my gentoo system) to see if any crashes still happen; and full time having youtube music playing(just as I did before in fact) in an attempt to give it opportunity to crash. All this inside a virtualbox gentoo guest OS, which I should probably mention that sometimes(kinda rarely) on boot manages to clock the audio driver wrongly and everything sounds slightly low pitched(or is it slowed too?):
Wrong:

snd_intel8x0 0000:00:05.0: clocking to 41131

Right:

snd_intel8x0 0000:00:05.0: clocking to 48000

But no crashes happened when it was clocked wrongly, because I would restart(shutdown,start) soon in order to fix it.

The gstavauddec.c version that I used was latest available (~amd64 and amd64 both point to it) media-plugins/gst-plugins-libav 1.4.5-r1 * for gentoo no-multilib (not hardened either) (default/linux/amd64/13.0/no-multilib) with kernel 4.2.0-rc4 (git)

  • there's no possibility to (easily?) use the git version of gst-plugins-libav (there's no -9999 version)

The ffmpeg version(commit 5bf8590) that I'm (still) testing (with those patches on top of it) is:
# ffmpeg -version
ffmpeg started on 2015-08-03 at 07:14:59
Report written to "ffmpeg-20150803-071459.log"
ffmpeg version N-74201-g5bf8590 Copyright (c) 2000-2015 the FFmpeg developers
built with gcc 5.1.0 (Gentoo 5.1.0 p1.2, pie-0.6.3)
configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --mandir=/usr/share/man --enable-shared --cc=x86_64-pc-linux-gnu-gcc --cxx=x86_64-pc-linux-gnu-g++ --ar=x86_64-pc-linux-gnu-ar --optflags=' ' --disable-static --enable-avfilter --enable-avresample --disable-stripping --disable-indev=v4l2 --disable-outdev=v4l2 --disable-indev=alsa --disable-indev=oss --disable-indev=jack --disable-outdev=alsa --disable-outdev=oss --disable-outdev=sdl --enable-bzlib --disable-runtime-cpudetect --disable-debug --disable-doc --disable-gnutls --enable-gpl --enable-hardcoded-tables --enable-iconv --disable-lzma --enable-network --disable-openssl --enable-postproc --disable-libsmbclient --disable-ffplay --disable-vaapi --disable-vdpau --enable-xlib --disable-libxcb --disable-libxcb-shm --disable-libxcb-xfixes --enable-zlib --disable-libcdio --disable-libiec61883 --disable-libdc1394 --disable-libcaca --disable-openal --disable-opengl --disable-libv4l2 --enable-libpulse --disable-libopencore-amrwb --disable-libopencore-amrnb --disable-libfdk-aac --disable-libopenjpeg --disable-libbluray --disable-libcelt --disable-libgme --disable-libgsm --disable-libmodplug --disable-libopus --disable-libquvi --disable-librtmp --disable-libssh --disable-libschroedinger --disable-libspeex --disable-libvorbis --disable-libvpx --disable-libzvbi --disable-libbs2b --disable-libflite --disable-frei0r --disable-libfribidi --disable-fontconfig --disable-ladspa --disable-libass --disable-libfreetype --disable-libsoxr --enable-pthreads --disable-libvo-aacenc --disable-libvo-amrwbenc --disable-libmp3lame --disable-libaacplus --disable-libfaac --disable-libsnappy --disable-libtheora --disable-libtwolame --disable-libwavpack --disable-libwebp --disable-libx264 --disable-libx265 --disable-libxvid --enable-x11grab --disable-avx --disable-avx2 --disable-fma3 --disable-fma4 --disable-ssse3 --disable-sse4 --disable-sse42 --disable-xop --cpu=host
libavutil 54. 29.100 / 54. 29.100
libavcodec 56. 56.101 / 56. 56.101
libavformat 56. 40.101 / 56. 40.101
libavdevice 56. 4.100 / 56. 4.100
libavfilter 5. 30.100 / 5. 30.100
libavresample 2. 1. 0 / 2. 1. 0
libswscale 3. 1.101 / 3. 1.101
libswresample 1. 2.101 / 1. 2.101
libpostproc 53. 3.100 / 53. 3.100

I'll give more details if a crash happens again, but honestly(from what I can tell so far, after 1 full day of no crashes) I'm quite optimistic that it won't. But who knows.
Thanks!

PS: I don't know what to do about the potential causes that you mentioned, michael, but I will consider them when crashes happen again. Thanks for looking into it more deeply.

comment:6 by zazdxscf, 9 years ago

Finally managed to make it crash. I figure it would never crash this way unless I do some compilation in the background to "poison" the memory or something(else?): I started compiling gcc 5.2.0 (just to have something to compile) and after like 10 minutes, firefox crashed in the same place.

I will attach backtrace_simple5.log (bt full) because it looks ugly if I just paste it here.

This is the used ffmpeg version:
$ ffmpeg -version
ffmpeg started on 2015-08-03 at 18:12:18
Report written to "ffmpeg-20150803-181218.log"
ffmpeg version N-74201-g5bf8590 Copyright (c) 2000-2015 the FFmpeg developers
built with gcc 5.1.0 (Gentoo 5.1.0 p1.2, pie-0.6.3)
configuration: --prefix=/usr --libdir=/usr/lib64 --shlibdir=/usr/lib64 --mandir=/usr/share/man --enable-shared --cc=x86_64-pc-linux-gnu-gcc --cxx=x86_64-pc-linux-gnu-g++ --ar=x86_64-pc-linux-gnu-ar --optflags=' ' --disable-static --enable-avfilter --enable-avresample --disable-stripping --disable-indev=v4l2 --disable-outdev=v4l2 --disable-indev=alsa --disable-indev=oss --disable-indev=jack --disable-outdev=alsa --disable-outdev=oss --disable-outdev=sdl --enable-bzlib --disable-runtime-cpudetect --disable-debug --disable-doc --disable-gnutls --enable-gpl --enable-hardcoded-tables --enable-iconv --disable-lzma --enable-network --disable-openssl --enable-postproc --disable-libsmbclient --disable-ffplay --disable-vaapi --disable-vdpau --enable-xlib --disable-libxcb --disable-libxcb-shm --disable-libxcb-xfixes --enable-zlib --disable-libcdio --disable-libiec61883 --disable-libdc1394 --disable-libcaca --disable-openal --disable-opengl --disable-libv4l2 --enable-libpulse --disable-libopencore-amrwb --disable-libopencore-amrnb --disable-libfdk-aac --disable-libopenjpeg --disable-libbluray --disable-libcelt --disable-libgme --disable-libgsm --disable-libmodplug --disable-libopus --disable-libquvi --disable-librtmp --disable-libssh --disable-libschroedinger --disable-libspeex --disable-libvorbis --disable-libvpx --disable-libzvbi --disable-libbs2b --disable-libflite --disable-frei0r --disable-libfribidi --disable-fontconfig --disable-ladspa --disable-libass --disable-libfreetype --disable-libsoxr --enable-pthreads --disable-libvo-aacenc --disable-libvo-amrwbenc --disable-libmp3lame --disable-libaacplus --disable-libfaac --disable-libsnappy --disable-libtheora --disable-libtwolame --disable-libwavpack --disable-libwebp --disable-libx264 --disable-libx265 --disable-libxvid --enable-x11grab --disable-avx --disable-avx2 --disable-fma3 --disable-fma4 --disable-ssse3 --disable-sse4 --disable-sse42 --disable-xop --cpu=host
libavutil 54. 29.100 / 54. 29.100
libavcodec 56. 56.101 / 56. 56.101
libavformat 56. 40.101 / 56. 40.101
libavdevice 56. 4.100 / 56. 4.100
libavfilter 5. 30.100 / 5. 30.100
libavresample 2. 1. 0 / 2. 1. 0
libswscale 3. 1.101 / 3. 1.101
libswresample 1. 2.101 / 1. 2.101
libpostproc 53. 3.100 / 53. 3.100

Thus the commit is 5bf8590 (titled: "avfilter/avf_showvolume: stop making output fully transparent")
and apply the two included patches from above to get the exact source code that was used in my ffmpeg version to which the backtrace log applies (to make sure the line numbers match)

But to make it easier I reiterate here the important ones:

#3 0x00007f77040262a1 in NEG_USR32 (s=<optimized out>, a=<optimized out>) at /usr/src/debug/media-video/ffmpeg-9999/ffmpeg-9999/libavcodec/x86/mathops.h:125

#define NEG_USR32 NEG_USR32
static inline uint32_t NEG_USR32(uint32_t a, int8_t s){
    __asm__ ("shrl %1, %0\n\t"  //<------ this is line 125
         : "+r" (a)
         : "ic" ((uint8_t)(-s))
    );
    return a;
}

#4 decode_spectrum_and_dequant (band_type=0x7f76c8181d7c, ics=0x7f76c8181100, pulse=0x7f76d0afcea0, pulse_present=0, sf=0x7f76c818215c, gb=0x7f76d0afd2a0, coef=0x7f76c81839c0, ac=0x7f76cc4f1000) at /usr/src/debug/media-video/ffmpeg-9999/ffmpeg-9999/libavcodec/aacdec_template.c:1681

do {
                            int code;
                            unsigned cb_idx;

                            UPDATE_CACHE(re, gb);
                            GET_VLC(code, re, gb, vlc_tab, 8, 2); //<---- line 1681 is this*
                            cb_idx = cb_vector_idx[code];
#if USE_FIXED
                            cf = DEC_SPAIR(cf, cb_idx);
#else
                            cf = VMUL2(cf, vq, cb_idx, sf + idx);
#endif /* USE_FIXED */
                        } while (len -= 2);
  • note here that in my initial post(up top) I am now unsure if it really crashed in the above(UPDATE_CACHE) line or if I actually used an older coredump with updated sources! So it might've been the GET_VLC line all the time! But, it seems that UPDATE_CACHE is called in GET_VLC too and it eventually calls that NEG_USR32 so it might've been the case that it did crash in those 2 different close-by places after all just because the both reach NEG_USR32 through UPDATE_CACHE.

#5 decode_ics (ac=ac@entry=0x7f76cc4f1000, sce=sce@entry=0x7f76c8181100, gb=gb@entry=0x7f76d0afd2a0, common_window=common_window@entry=1, scale_flag=0) at /usr/src/debug/media-video/ffmpeg-9999/ffmpeg-9999/libavcodec/aacdec_template.c:1958

    if (decode_spectrum_and_dequant(ac, out, gb, sce->sf, pulse_present, //<---- this be line 1958
                                    &pulse, ics, sce->band_type) < 0)
        return AVERROR_INVALIDDATA;

#6 0x00007f7704026e1c in decode_cpe (ac=ac@entry=0x7f76cc4f1000, gb=gb@entry=0x7f76d0afd2a0, cpe=cpe@entry=0x7f76c8173000) at /usr/src/debug/media-video/ffmpeg-9999/ffmpeg-9999/libavcodec/aacdec_template.c:2084

    if ((ret = decode_ics(ac, &cpe->ch[0], gb, common_window, 0)))
        return ret;
    if ((ret = decode_ics(ac, &cpe->ch[1], gb, common_window, 0))) //<---- this be line 2084
        return ret;

#7 0x00007f7704027cd8 in aac_decode_frame_int (avctx=avctx@entry=0x7f76e9a6fe00, data=data@entry=0x7f76d0afd4f0, got_frame_ptr=got_frame_ptr@entry=0x7f76d0afd868, gb=gb@entry=0x7f76d0afd2a0, avpkt=avpkt@entry=0x7f76d0afd350) at /usr/src/debug/media-video/ffmpeg-9999/ffmpeg-9999/libavcodec/aacdec_template.c:2959

        case TYPE_CPE:
            err = decode_cpe(ac, gb, che); //<--- this be line 2959
            audio_found = 1;
            break;

#8 0x00007f7704028cfa in aac_decode_frame (avctx=0x7f76e9a6fe00, data=0x7f76d0afd4f0, got_frame_ptr=0x7f76d0afd868, avpkt=0x7f76d0afd350) at /usr/src/debug/media-video/ffmpeg-9999/ffmpeg-9999/libavcodec/aacdec_template.c:3136

    default:
        err = aac_decode_frame_int(avctx, data, got_frame_ptr, &gb, avpkt); //<---- this is line 3136
    }

#9 0x00007f77043b8501 in avcodec_decode_audio4 (avctx=0x7f76e9a6fe00, frame=frame@entry=0x7f76d0afd4f0, got_frame_ptr=got_frame_ptr@entry=0x7f76d0afd868, avpkt=avpkt@entry=0x7f76d0afd420) at /usr/src/debug/media-video/ffmpeg-9999/ffmpeg-9999/libavcodec/utils.c:2597

        else {
            ret = avctx->codec->decode(avctx, frame, got_frame_ptr, &tmp); //<--- this be line 2597
            av_assert0(ret <= tmp.size);
            frame->pkt_dts = avpkt->dts;
        }

#10 0x00007f77054d806d in gst_ffmpegauddec_audio_frame (ffmpegdec=ffmpegdec@entry=0x7f770c5fc2a0, data=data@entry=0x7f76caefff00 "!\032T\375\266\217\003R\233Hʩ\300*\n\216\005\205gF\242\301\352\260%\250\375M\230\063\371\"\026\260\203\350Y\302<~߹ם\353!q]\227\311\031\350\231@饫1uv:\314z\251\223{\034\373l\205\364k\357ژ\034036\017P\210 &9\334)\221\004\204\230\217{Jq\310\004.\254\230(\216\060(\230B
\210s\337\060\216v\250\223R\263\033\267SzkQ\027\274\362\231\257is\300gp\332\327\336/u\021%s\003\246\246\262\362@\311\022\247\005\064\355\367\362\231D\252i\222\025\003\267\065\211\n\222\030\301aTC\234\224\230\232\340D\230\222\205\025\271R\022K(\250G:+\205h)\341\375M\023\266", <incomplete sequence \363\207>..., size=<optimized out>, have_data=have_data@entry=0x7f76d0afd868, outbuf=outbuf@entry=0x7f76d0afd7d0, ret=ret@entry=0x7f76d0afd86c, in_plugin=<optimized out>) at /usr/src/debug/media-plugins/gst-plugins-libav-1.4.5-r1/gst-libav-1.4.5/ext/libav/gstavauddec.c:475

static gint
gst_ffmpegauddec_audio_frame (GstFFMpegAudDec * ffmpegdec,
    AVCodec * in_plugin, guint8 * data, guint size, gint * have_data,
    GstBuffer ** outbuf, GstFlowReturn * ret)
{
  gint len = -1;
  AVPacket packet;
  AVFrame frame;

  GST_DEBUG_OBJECT (ffmpegdec, "size: %d", size);

  gst_avpacket_init (&packet, data, size);
  memset (&frame, 0, sizeof (frame));
  avcodec_get_frame_defaults (&frame);
  len = avcodec_decode_audio4 (ffmpegdec->context, &frame, have_data, &packet); // <--- this be line 475

  GST_DEBUG_OBJECT (ffmpegdec,
      "Decode audio: len=%d, have_data=%d", len, *have_data);
...

#11 0x00007f77054d8622 in gst_ffmpegauddec_frame (ffmpegdec=ffmpegdec@entry=0x7f770c5fc2a0, data=data@entry=0x7f76caefff00 "!\032T\375\266\217\003R\233Hʩ\300*\n\216\005\205gF\242\301\352\260%\250\375M\230\063\371\"\026\260\203\350Y\302<~߹ם\353!q]\227\311\031\350\231@饫1uv:\314z\251\223{\034\373l\205\364k\357ژ\034\036\07P\210 &9\334)\221\004\204\230\217{Jq\310\004.\254\230(\216\060(\230B
\210s\337\060\216v\250\223R\263\033\267SzkQ\027\274\362\231\257is\300gp\332\327\336/u\021%s\003\246\246\262\362@\311\022\247\005\064\355\367\362\231D\252i\222\025\003\267\065\211\n\222\030\301aTC\234\224\230\232\340D\230\222\205\025\271R\022K(\250G:+\205h)\341\375M\023\266", <incomplete sequence \363\207>..., size=size@entry=256, have_data=have_data@entry=0x7f76d0afd868, ret=ret@entry=0x7f76d0afd86c) at /usr/src/debug/media-plugins/gst-plugins-libav-1.4.5-r1/gst-libav-1.4.5/ext/libav/gstavauddec.c:632

  *ret = GST_FLOW_OK;
  ffmpegdec->context->frame_number++;

  oclass = (GstFFMpegAudDecClass *) (G_OBJECT_GET_CLASS (ffmpegdec));

  len = //<---- this be line 632
      gst_ffmpegauddec_audio_frame (ffmpegdec, oclass->in_plugin, data, size,
      have_data, &outbuf, ret);

...

Linux norm2 4.2.0-rc4-g45b4b78 #3 SMP Wed Jul 29 13:39:07 CEST 2015 x86_64 AMD A6-3400M APU with Radeon(tm) HD Graphics AuthenticAMD GNU/Linux
This gentoo no-multilib(and not hardened) which is running inside a virtualbox
firefox version is 39.0

If you have any suggestions on what I should try next, I'd be more than happy to. Even if it's about code in gst-plugins-libav ... or anything really. (I don't know much btw, but willing to try)

by zazdxscf, 9 years ago

Attachment: backtrace_simple5.log​ added

gdb bt full with commit 5bf8590d6e and the 2 patches on top of it

in reply to:  6 comment:7 by Michael Niedermayer, 9 years ago

Replying to zazdxscf:

#10 0x00007f77054d806d in gst_ffmpegauddec_audio_frame (ffmpegdec=ffmpegdec@entry=0x7f770c5fc2a0, 

> static gint
> gst_ffmpegauddec_audio_frame (GstFFMpegAudDec * ffmpegdec,
>     AVCodec * in_plugin, guint8 * data, guint size, gint * have_data,
>     GstBuffer ** outbuf, GstFlowReturn * ret)
> {
>   gint len = -1;
>   AVPacket packet;
>   AVFrame frame;
> 
>   GST_DEBUG_OBJECT (ffmpegdec, "size: %d", size);
> 
>   gst_avpacket_init (&packet, data, size);
>   memset (&frame, 0, sizeof (frame));
>   avcodec_get_frame_defaults (&frame);
>   len = avcodec_decode_audio4 (ffmpegdec->context, &frame, have_data, &packet); // <--- this be line 475
> 
>   GST_DEBUG_OBJECT (ffmpegdec,
>       "Decode audio: len=%d, have_data=%d", len, *have_data);
> ...

[...]

If you have any suggestions on what I should try next, I'd be more than happy to. Even if it's about code in gst-plugins-libav ... or anything really. (I don't know much btw, but willing to try)

The fix for this is likely this commit in gstreamer:
​http://cgit.freedesktop.org/gstreamer/gst-libav/commit/?id=30a4a28793f2e0ff08aaea368b7c14317ac2ca21

There seem to be other related fixes in gstreamer too
i dont think theres any sense in debuging this further before you ensured that your gstreamer contains these fixes

comment:8 by zazdxscf, 9 years ago

Roger that, I'll try to bring gstreamer up to date(somehow) and then report back IF/when another crash occurs. Thank you!

comment:9 by Michael Niedermayer, 9 years ago

Resolution: → invalid
Status: new → closed

ok, please reopen if you can still reproduce with a updated gstreamer, but i think it should then work fine

Note: See TracTickets for help on using tickets.