libutvideo fails to link to FFmpeg if assembly is enabled

[qyot27]

If libutvideo is compiled with assembly enabled (from ​this commit), then FFmpeg fails to link to it with undefined references to the assembly stuff:

MAN	doc/ffserver.1
TXT	doc/fate.txt
CC	cmdutils.o
CC	ffmpeg.o
LD	ffmpeg_g
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xa0): undefined reference to `x86_sse2_PredictMedianAndCount_align16'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xa4): undefined reference to `x86_sse2_PredictMedianAndCount_align1'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xa8): undefined reference to `x86_sse2_PredictLeftAndCount_align1'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xac): undefined reference to `x86_sse1mmx_RestoreMedian_align1'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xb0): undefined reference to `x86_i686_HuffmanEncode'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xb4): undefined reference to `x86_i686_HuffmanDecode'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xb8): undefined reference to `x86_i686_HuffmanDecodeAndAccum'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xbc): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep2'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xc0): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xc4): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForBottomupRGB32Green'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xc8): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForBottomupRGB32Blue'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xcc): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForBottomupRGB32Red'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xd0): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForBottomupRGB32RedAndDummyAlpha'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xd4): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep3ForBottomupRGB24Green'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xd8): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep3ForBottomupRGB24Blue'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xdc): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep3ForBottomupRGB24Red'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xe0): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForTopdownRGB32Green'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xe4): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForTopdownRGB32Blue'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xe8): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForTopdownRGB32Red'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xec): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForTopdownRGB32RedAndDummyAlpha'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xf0): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep3ForTopdownRGB24Green'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xf4): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep3ForTopdownRGB24Blue'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xf8): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep3ForTopdownRGB24Red'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0xfc): undefined reference to `x86_sse2_ConvertULY2ToBottomupRGB24'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x100): undefined reference to `x86_sse2_ConvertULY2ToBottomupRGB32'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x104): undefined reference to `x86_sse2_ConvertULY2ToTopdownRGB24'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x108): undefined reference to `x86_sse2_ConvertULY2ToTopdownRGB32'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x10c): undefined reference to `x86_sse2_ConvertBottomupRGB24ToULY2'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x110): undefined reference to `x86_sse2_ConvertBottomupRGB32ToULY2'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x114): undefined reference to `x86_sse2_ConvertTopdownRGB24ToULY2'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x118): undefined reference to `x86_sse2_ConvertTopdownRGB32ToULY2'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x12c): undefined reference to `x86_i686_RestoreMedian_align1'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x130): undefined reference to `x86_i686_HuffmanEncode'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x134): undefined reference to `x86_i686_HuffmanDecode'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x138): undefined reference to `x86_i686_HuffmanDecodeAndAccum'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x13c): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep2'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x140): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x144): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForBottomupRGB32Green'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x148): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForBottomupRGB32Blue'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x14c): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForBottomupRGB32Red'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x150): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForBottomupRGB32RedAndDummyAlpha'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x154): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep3ForBottomupRGB24Green'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x158): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep3ForBottomupRGB24Blue'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x15c): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep3ForBottomupRGB24Red'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x160): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForTopdownRGB32Green'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x164): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForTopdownRGB32Blue'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x168): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForTopdownRGB32Red'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x16c): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep4ForTopdownRGB32RedAndDummyAlpha'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x170): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep3ForTopdownRGB24Green'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x174): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep3ForTopdownRGB24Blue'
/usr/local/lib/libutvideo.a(TunedFunc.o):(.rodata+0x178): undefined reference to `x86_i686_HuffmanDecodeAndAccumStep3ForTopdownRGB24Red'
collect2: ld returned 1 exit status
make: *** [ffmpeg_g] Error 1

Relevant configuration info:
$ gcc --version
gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3

$ nasm -v
NASM version 2.09.10 compiled on Oct 17 2011

FFmpeg N-43331-gee475e2:
./configure --prefix=$HOME/ffmpeg_build --enable-gpl --enable-version3 --enable-libutvideo

[dbuitenh]

That's not official upstream, FWIW. I hacked that in. Are you sure you set the proper format for it with OS= and ARCH=? Check the format of the files NASM is producing.

[qyot27]

According to the output of file, the object files NASM produced seem to be what they're supposed to be (ELF 32-bit if I don't specify OS= and use ARCH=x86 and COFF if I make sure OS=windows and ARCH=x86).

I was working with my own ​buildsystem branch, which I had integrated that commit into. I also went back and double-checked with your's to make sure the files produced match the same way.

After some testing, if I was compiling for my 64-bit setup and enabled x64 asm, then everything linked (even with my branch), although it resulted in a segfault whenever libutvideo was used for decoding or encoding. The linking problems seem to be focused only on compiling libutvideo+asm and ffmpeg as 32-bit.

Would changing elif __i386__ to elif __x86_32__ in TunedFunc.cpp actually work? I thought about it because it would parallel the x64 pattern right above it, since that one does link (segfault notwithstanding). It seemed to let a 32-bit compile link to ffmpeg, but for all I know I just told it to ignore assembly altogether (even though running nm -s on the resultant libutvideo.a showed the *asm_x86.o files and their symbols). The 32-bit build of libutvideo+asm/ffmpeg also did not segfault the way the 64-bit one did, so I'm unsure if that's also hinting to it not actually having the assembly linked in or simply that the x64 asm has a problem (it did show some warnings while compiling, or maybe it was because I forgot to use PIC).

And while this is somewhat off-topic, the define being -DSTATIC_LIB_WITH_ASM, does that preclude the ability of shared libraries using it? I would assume it does, but I just want to be sure. I have shared compilation worked out in my branch (although I've still not hammered all the kinks out of the install step or having it build alongside the static lib), and FFmpeg seems to work with it without problems, but I'm not sure how the STATIC_LIB_WITH_ASM thing would factor in, if at all.

[Carl Eugen Hoyos]

Works fine here with git://github.com/qyot27/libutvideo