Changes between Initial Version and Version 1 of Create a mosaic out of several input videos using xstack


Ignore:
Timestamp:
Oct 8, 2019, 7:42:41 PM (2 months ago)
Author:
jpb
Comment:

initial page edit

Legend:

Unmodified
Added
Removed
Modified
  • Create a mosaic out of several input videos using xstack

    v1 v1  
     1
     2[[PageOutline(1-100,Contents)]]
     3
     4= Overview =
     5
     6FFmpeg provides more than one mechanism to create a video mosaic.  In addition to using the ''overlay'' filter shown in  [[Create a mosaic out of several input videos]], you can use the [http://ffmpeg.org/ffmpeg-all.html#xstack xstack] filter.  [http://ffmpeg.org/ffmpeg-all.html#xstack xstack] takes the individual inputs along with a custom layout that you create, and positions the videos according to the layout.  Here's an example:
     7
     8[[Image(col_2x2.png, title=Figure 1)]]
     9
     10
     11As shown in the documentation [http://ffmpeg.org/ffmpeg-all.html#xstack xstack] displays video streams in [https://en.wikipedia.org/wiki/Row-_and_column-major_order Column Major Order], but you can change the layout to achieve '''Row Major Order'''.  Examples of both are shown below.
     12
     13[http://ffmpeg.org/ffmpeg-all.html#xstack xstack] uses a complex filter similar to overlay.  The major difference is that there is no need to use a nullsrc filter and the overlay filter.  [http://ffmpeg.org/ffmpeg-all.html#xstack xstack] figures out how to position the videos according to your layout specification.  We'll explain the layout specification shortly.  Here is the command line used to create the above video mosaic (shown with Unix / Linux line continuation using the '\' character):
     14
     15{{{
     16ffmpeg \
     17   -i videos/01.mkv \
     18   -i videos/02.mkv \
     19   -i videos/03.mkv \
     20   -i videos/04.mkv \
     21  -filter_complex " \
     22      [0:v] setpts=PTS-STARTPTS, scale=qvga [a0]; \
     23      [1:v] setpts=PTS-STARTPTS, scale=qvga [a1]; \
     24      [2:v] setpts=PTS-STARTPTS, scale=qvga [a2]; \
     25      [3:v] setpts=PTS-STARTPTS, scale=qvga [a3]; \
     26      [a0][a1][a2][a3]xstack=inputs=4:layout=0_0|0_h0|w0_0|w0_h0[out] \
     27      " \
     28    -map "[out]" \
     29    -c:v libx264 -t '30' -f matroska output_col_2x2.mkv
     30}}}
     31
     32This example shows four videos in a two by two matrix. As described in the [http://ffmpeg.org/ffmpeg-all.html#xstack xstack layout documentation] the xstack line contains:
     33
     34
     35* a list of input labels  ('''[a0] [a1] [a2] [a3]''') as shown above. The label name is not important and could be descriptive of the input (e.g. '''[streetview]'''), however each label must match the input stream label above.
     36
     37* the xstack command ('''xstack=''')
     38
     39* an input quantity specification ('''inputs=4:''') The number must match the total number of input streams, or the '''ffmpeg''' command will throw an error
     40
     41* a layout specification ('''layout=0_0|0_h0|w0_0|w0_h0'''), a series of layout descriptors separated by the pipe symbol ' '''|''' '.  Each descriptor refers to width and  height separated by the underscore ' '''_''' '. In these descriptors, width is always on the left of the underscore and height is always on the right.  These correspond to (x,y) coordinates as described below.
     42
     43* a map label ('''[out]''') which is used by the -map option
     44
     45Here is the meaning of each layout descriptor and a graphic shown in Figure 2, keeping in mind that this description is for '''Column Major Order'''.
     46
     47[[Image(layout_col_2x2.png, title=Figure 2)]]
     48
     49
     50'''0_0''' describes the point of origin for the first input stream ('''[a0]'''). Since there is no 'w' or 'h' with this descriptor, the point describes where to place the first input stream - i.e. the top left of the mosaic matrix at coordinates (0,0) relative to the display window.
     51
     52'''0_h0''' describes the point of origin for the second stream ('''[a1]''').  This is read as '' "Use the same X value as stream '''[a0]''' and use the Y value that is the height of '''[a0]'''. This places the point at (0, height_of_[a0])." ''
     53
     54'''w0_0''' describes the point of origin for the third stream ('''[a2]''').  This is read as '' "Use the X value as the width of the first stream ('''[a0]''') and the y value as  0, or (width_of_[a0], 0)." ''
     55
     56'''w0_h0''' describes the point of origin for the fourth stream ('''[a3]''').  '' "Use the X and Y values as the width and height of the first stream ('''[a0]'''), with the result of (width_of_[0],height_of_[a0])." ''
     57
     58When there are more than two images in a sequence (either vertical or horizontal), it becomes necessary to add widths or heights together.  So for example, for a single vertical sequence of three videos in '''Column Major Order''':
     59
     60[[Image(col_1x3.png, title=Figure 3)]]
     61
     62the layout would be '''layout=0_0|0_h0|0_h0+h1'''.  The X coordinate for all videos is the same - '''0''', whereas the Y coordinate changes.  For the first stream it is '''0''', then '''h0''', then '''h0+h1''' - the addition of height h0 and h1.  For a horizontal sequence it would be similar with identical Y coordinates and X coordinates of '''0''', '''w0''', and '''w0+w1'''.
     63
     64'''Row Major Order''' is similar, with values juxtaposed to appropriate points of origin.  Figure 4 shows a 3x3 matrix (nine video streams), and Figure 5 shows a 6x6 matrix (36 video streams).  Note how in these examples the output is now piped directly to '''ffplay'''.  (Be aware that higher video counts can result in higher than normal CPU load.)
     65
     66[[Image(row_3x3.png, title=Figure 4)]]
     67
     68The complete command for this example ('''Row Major Order''') is
     69{{{
     70ffmpeg \
     71   -i videos/01.mkv \
     72   -i videos/02.mkv \
     73   -i videos/03.mkv \
     74   -i videos/04.mkv \
     75   -i videos/05.mkv \
     76   -i videos/06.mkv \
     77   -i videos/07.mkv \
     78   -i videos/08.mkv \
     79   -i videos/09.mkv \
     80  -filter_complex " \
     81      [0:v] setpts=PTS-STARTPTS, scale=qvga [a0]; \
     82      [1:v] setpts=PTS-STARTPTS, scale=qvga [a1]; \
     83      [2:v] setpts=PTS-STARTPTS, scale=qvga [a2]; \
     84      [3:v] setpts=PTS-STARTPTS, scale=qvga [a3]; \
     85      [4:v] setpts=PTS-STARTPTS, scale=qvga [a4]; \
     86      [5:v] setpts=PTS-STARTPTS, scale=qvga [a5]; \
     87      [6:v] setpts=PTS-STARTPTS, scale=qvga [a6]; \
     88      [7:v] setpts=PTS-STARTPTS, scale=qvga [a7]; \
     89      [8:v] setpts=PTS-STARTPTS, scale=qvga [a8]; \
     90      [a0][a1][a2][a3][a4][a5][a6][a7][a8]xstack=inputs=9:layout=0_0|w0_0|w0+w1_0|0_h0|w0_h0|w0+w1_h0|0_h0+h1|w0_h0+h1|w0+w1_h0+h1[out] \
     91      " \
     92    -map "[out]" \
     93    -c:v libx264 -t '30' -f matroska -  | ffplay -autoexit  -left 30 -top 30  -
     94}}}
     95
     96
     97[[Image(row_6x6.png, title=Figure 5)]]
     98
     99For higher density mosaics, you should probably scale down all inputs.  This example scaled each video to qqvga (160x120).   
     100
     101The complete command for this example ('''Row Major Order''') is:
     102
     103{{{
     104ffmpeg \
     105   -i videos/01.mkv \
     106   -i videos/02.mkv \
     107   -i videos/03.mkv \
     108   -i videos/04.mkv \
     109   -i videos/05.mkv \
     110   -i videos/06.mkv \
     111   -i videos/07.mkv \
     112   -i videos/08.mkv \
     113   -i videos/09.mkv \
     114   -i videos/10.mkv \
     115   -i videos/11.mkv \
     116   -i videos/12.mkv \
     117   -i videos/13.mkv \
     118   -i videos/14.mkv \
     119   -i videos/15.mkv \
     120   -i videos/16.mkv \
     121   -i videos/17.mkv \
     122   -i videos/18.mkv \
     123   -i videos/19.mkv \
     124   -i videos/20.mkv \
     125   -i videos/21.mkv \
     126   -i videos/22.mkv \
     127   -i videos/23.mkv \
     128   -i videos/24.mkv \
     129   -i videos/25.mkv \
     130   -i videos/26.mkv \
     131   -i videos/27.mkv \
     132   -i videos/28.mkv \
     133   -i videos/29.mkv \
     134   -i videos/30.mkv \
     135   -i videos/31.mkv \
     136   -i videos/32.mkv \
     137   -i videos/33.mkv \
     138   -i videos/34.mkv \
     139   -i videos/35.mkv \
     140   -i videos/36.mkv \
     141  -filter_complex " \
     142      [0:v] setpts=PTS-STARTPTS, scale=qqvga [a0]; \
     143      [1:v] setpts=PTS-STARTPTS, scale=qqvga [a1]; \
     144      [2:v] setpts=PTS-STARTPTS, scale=qqvga [a2]; \
     145      [3:v] setpts=PTS-STARTPTS, scale=qqvga [a3]; \
     146      [4:v] setpts=PTS-STARTPTS, scale=qqvga [a4]; \
     147      [5:v] setpts=PTS-STARTPTS, scale=qqvga [a5]; \
     148      [6:v] setpts=PTS-STARTPTS, scale=qqvga [a6]; \
     149      [7:v] setpts=PTS-STARTPTS, scale=qqvga [a7]; \
     150      [8:v] setpts=PTS-STARTPTS, scale=qqvga [a8]; \
     151      [9:v] setpts=PTS-STARTPTS, scale=qqvga [a9]; \
     152      [10:v] setpts=PTS-STARTPTS, scale=qqvga [a10]; \
     153      [11:v] setpts=PTS-STARTPTS, scale=qqvga [a11]; \
     154      [12:v] setpts=PTS-STARTPTS, scale=qqvga [a12]; \
     155      [13:v] setpts=PTS-STARTPTS, scale=qqvga [a13]; \
     156      [14:v] setpts=PTS-STARTPTS, scale=qqvga [a14]; \
     157      [15:v] setpts=PTS-STARTPTS, scale=qqvga [a15]; \
     158      [16:v] setpts=PTS-STARTPTS, scale=qqvga [a16]; \
     159      [17:v] setpts=PTS-STARTPTS, scale=qqvga [a17]; \
     160      [18:v] setpts=PTS-STARTPTS, scale=qqvga [a18]; \
     161      [19:v] setpts=PTS-STARTPTS, scale=qqvga [a19]; \
     162      [20:v] setpts=PTS-STARTPTS, scale=qqvga [a20]; \
     163      [21:v] setpts=PTS-STARTPTS, scale=qqvga [a21]; \
     164      [22:v] setpts=PTS-STARTPTS, scale=qqvga [a22]; \
     165      [23:v] setpts=PTS-STARTPTS, scale=qqvga [a23]; \
     166      [24:v] setpts=PTS-STARTPTS, scale=qqvga [a24]; \
     167      [25:v] setpts=PTS-STARTPTS, scale=qqvga [a25]; \
     168      [26:v] setpts=PTS-STARTPTS, scale=qqvga [a26]; \
     169      [27:v] setpts=PTS-STARTPTS, scale=qqvga [a27]; \
     170      [28:v] setpts=PTS-STARTPTS, scale=qqvga [a28]; \
     171      [29:v] setpts=PTS-STARTPTS, scale=qqvga [a29]; \
     172      [30:v] setpts=PTS-STARTPTS, scale=qqvga [a30]; \
     173      [31:v] setpts=PTS-STARTPTS, scale=qqvga [a31]; \
     174      [32:v] setpts=PTS-STARTPTS, scale=qqvga [a32]; \
     175      [33:v] setpts=PTS-STARTPTS, scale=qqvga [a33]; \
     176      [34:v] setpts=PTS-STARTPTS, scale=qqvga [a34]; \
     177      [35:v] setpts=PTS-STARTPTS, scale=qqvga [a35]; \
     178      [a0][a1][a2][a3][a4][a5][a6][a7][a8][a9][a10][a11][a12][a13][a14][a15][a16][a17][a18][a19][a20][a21][a22][a23][a24][a25][a26][a27][a28][a29][a30][a31][a32][a33][a34][a35]xstack=inputs=36:layout=0_0|w0_0|w0+w1_0|w0+w1+w2_0|w0+w1+w2+w3_0|w0+w1+w2+w3+w4_0|0_h0|w0_h0|w0+w1_h0|w0+w1+w2_h0|w0+w1+w2+w3_h0|w0+w1+w2+w3+w4_h0|0_h0+h1|w0_h0+h1|w0+w1_h0+h1|w0+w1+w2_h0+h1|w0+w1+w2+w3_h0+h1|w0+w1+w2+w3+w4_h0+h1|0_h0+h1+h2|w0_h0+h1+h2|w0+w1_h0+h1+h2|w0+w1+w2_h0+h1+h2|w0+w1+w2+w3_h0+h1+h2|w0+w1+w2+w3+w4_h0+h1+h2|0_h0+h1+h2+h3|w0_h0+h1+h2+h3|w0+w1_h0+h1+h2+h3|w0+w1+w2_h0+h1+h2+h3|w0+w1+w2+w3_h0+h1+h2+h3|w0+w1+w2+w3+w4_h0+h1+h2+h3|0_h0+h1+h2+h3+h4|w0_h0+h1+h2+h3+h4|w0+w1_h0+h1+h2+h3+h4|w0+w1+w2_h0+h1+h2+h3+h4|w0+w1+w2+w3_h0+h1+h2+h3+h4|w0+w1+w2+w3+w4_h0+h1+h2+h3+h4[out] \
     179      " \
     180    -map "[out]" \
     181    -c:v libx264 -t '30' -f matroska -  | ffplay -autoexit -left 10 -top 10  -
     182}}}
     183
     184Finally, you should note that if inputs are of different sizes, gaps or overlaps may occur. In such cases, you will have to scale the inputs individually.
     185
     186= Code =
     187
     188The code that created these scripts and files is available (BSD license) at https://www.jimby.name/techbits/recent/xstack/ .
     189
     190= Kudos =
     191
     192Kudos to the [http://git.videolan.org/?p=ffmpeg.git;a=blob_plain;f=MAINTAINERS;hb=HEAD great people behind FFmpeg project].   [http://www.ffmpeg.org/donations.html Donate to FFmpeg project here]
     193
     194= Matrix Madness! =
     195
     196[[Image(row_12x12.png)]]
     197
     198
     199
     200