Adding a FATE Test
There are two main kinds of FATE test: using existing samples (primarily useful with decoder testing), and using an artificially generated sample for testing.
The recipes for the tests written in GNU Make are located under
tests/fate in the FFmpeg source, and the tests themselves are executed through a shell wrapper script
tests/fate-run.sh. For the most part you do not need to know anything about the shell script for this wiki page, but we assume moderate knowledge and experience with GNU Make for this wiki page.
Because FFmpeg has a very complex codebase, differing code needs a wide variety of tests. Let us start with the common part of the tests. Then, after that, we'll have some notes on testing specific component like filters or decoders.
Three Parts of a Test
A typical test usually consists of four parts:
- Registration: registering your new test to
- Dependencies: which are usually samples for a format, a generated sample, or a program specifically written to test this feature;
- Configuration: this part is the most important of all, because it controls how the test will be run.
One Test for One Feature
The most common case is that only one test is allocated for one feature (e.g. decoder, muxer, hashing algorithm, etc.). Let's open
tests/fate/audio.mak, which contains tests for audio codecs, and scroll down to a test named
fate-dts. For convenience, you can also see a copy of the recipe below.
FATE_SAMPLES_AUDIO-$(call DEMDEC, MPEGTS, DCA) += fate-dts fate-dts: CMD = pcm -i $(TARGET_SAMPLES)/dts/dts.ts fate-dts: CMP = oneoff fate-dts: REF = $(SAMPLES)/dts/dts.pcm
This is one of the simplest tests for FFmpeg. Look at the first quoted line. It adds
fate-dts to a weird variable containing a call to
DEMDEC function returns "yes" if the required
DECoder are enabled, in this case the
mpegts demuxer and
dca decoder. So
fate-dts, the target name of the test, gets added into
Then, if you look at the end of the file:
FATE_SAMPLES_AUDIO += $(FATE_SAMPLES_AUDIO-yes) FATE_SAMPLES_FFMPEG += $(FATE_SAMPLES_AUDIO)
The enabled tests are added into
FATE_SAMPLES_AUDIO and subsequently
FATE_SAMPLES_FFMPEG variable, which then of course gets called when doing
Using this kind of system guarantees that if a person builds FFmpeg with certain parts disabled, the FATE test will skip the disabled parts.
Keep in mind that different tests may have different variable prefixes. Not all tests can be categorized into
<= 3 Tests for One Feature
Now knowing how to add one test for one feature, it is easy to add multiple tests. For example, you can just add multiple tests like this:
FATE_SAMPLES_AUDIO-$(call DEMDEC, MPEGTS, DCA) += fate-dts-1 fate-dts-1: CMD = pcm -i $(TARGET_SAMPLES)/dts/dts-1.ts fate-dts-1: CMP = oneoff fate-dts-1: REF = $(SAMPLES)/dts/dts-1.pcm FATE_SAMPLES_AUDIO-$(call DEMDEC, MPEGTS, DCA) += fate-dts-2 fate-dts-2: CMD = pcm -i $(TARGET_SAMPLES)/dts/dts-2.ts fate-dts-2: CMP = oneoff fate-dts-2: REF = $(SAMPLES)/dts/dts-2.pcm
However, this approach has some disadvantages:
- The long
FATE_SAMPLES_AUDIO-$(call DEMDEC, MPEGTS, DCA)is repeated for each test, which reduces readability.
- It is not possible to do
make fate-dtswhich runs these two tests (of course you can add another target that depends on the two tests, but that is not pretty for code readers).
The two disadvantages become more severe with more tests for this approach.
Lots of Tests for One Feature -- Irregular File Names
Therefore, you can use an additional variable that contains the targets related to a feature, and then use it.
tests/fate/amrnb.mk, which is a great example of multiple tests with irregular file names. (A part of the file is omitted for clarity below.)
FATE_AMRNB += fate-amrnb-4k75 fate-amrnb-4k75: CMD = pcm -i $(TARGET_SAMPLES)/amrnb/4.75k.amr fate-amrnb-4k75: REF = $(SAMPLES)/amrnb/4.75k.pcm FATE_AMRNB += fate-amrnb-5k15 fate-amrnb-5k15: CMD = pcm -i $(TARGET_SAMPLES)/amrnb/5.15k.amr fate-amrnb-5k15: REF = $(SAMPLES)/amrnb/5.15k.pcm FATE_AMRNB += fate-amrnb-5k9 fate-amrnb-5k9: CMD = pcm -i $(TARGET_SAMPLES)/amrnb/5.9k.amr fate-amrnb-5k9: REF = $(SAMPLES)/amrnb/5.9k.pcm [...] $(FATE_AMRNB): CMP = stddev FATE_SAMPLES_AVCONV-$(call DEMDEC, AMR, AMRNB) += $(FATE_AMRNB) fate-amrnb: $(FATE_AMRNB)
So here, first the target names of three tests are added to a variable
FATE_AMRNB, and then the
It is very important for you to declare the dependencies of a test if it uses any external files, like special samples or synthetic files.
This controls how the test will be run, the parameters of each test and so on. There exist many commands, the basic commands are:
This command first encodes an input file and then decodes the encoded file. It’s useful to test a audio encoder. An example of this command is fate-vorbis-encode test, which can be found in
tests/fate/vorbis.mak. The code of this command is the following:
FATE_VORBIS += fate-vorbis-encode fate-vorbis-encode: CMD = enc_dec_pcm ogg wav s16le $(TARGET_SAMPLES)/audio-reference/luckynight_2ch_44kHz_s16.wav -c:a vorbis -strict experimental fate-vorbis-encode: REF = $(SAMPLES)/audio-reference/luckynight_2ch_44kHz_s16.wav fate-vorbis-encode: CMP_SHIFT = 0 fate-vorbis-encode: CMP_TARGET = 2000 fate-vorbis-encode: SIZE_TOLERANCE = 12123 fate-vorbis-encode: FUZZ = 0 [...] fate-vorbis-encode: CMP = stddev
- enc_dec_pcm: indicates the command to be executed.
- ogg: indicates the format in which to encode the input file
- wav: indicates the format in which to decode the encoded file.
- s16le: indicates the codec_name.
- file: the input file
- @: the other commands to be included
- REF: indicates the reference file, the decoded file will be compared with this file.
- CMP_SHIFT: to calculate the CMP_SHIFT value you can use the tests/tini_psnr command. A guide how to use it is explained here.
- CMP_TARGET: it depends on what “CMP= “ is set.
- FUZZ: is the difference that is considered ok between the CMP_TARGET and the actual value.
- SIZE_TOLERANCE is the difference between the input and output sizes. The value of this parameter is calculated by subtracting the size of the two files.
To do a new test do the following:
- Compute CMP_SHIFT value with tiny_psnr. The program needs the encoder input and decoder output (that is 2 pcm wav or raw files).
- Copy the fate test to the .mak file and adjust the CMD options to match your needs.
- Run the new test, it should show errors like:
stddev: 296.37 PSNR: 46.89 MAXDIFF: 4810 bytes: 1675800/ 1679360 stddev: |296.37 - 2000| >= 0
- The CMP_TARGET for your test is 296 and your FUZZ is 0.
- The SIZE_TOLERANCE is 1679360 - 1675800 = 3560.
This page is licensed under Creative Commons Attribution-ShareAlike 2.0 Generic, 2.5 Generic, 3.0 Unported, or 4.0 International license.