Incorrect handling of "mbedtls_x509_crt_parse_file" return code

[Skybound]

Summary of the bug:

When using mbed TLS and loading a CA file, loading will fail if any of the certificates in the file cannot be parsed by mbed. For example, when having a name constraint as a critical extension which is not currently supported by mbed (​https://github.com/Mbed-TLS/mbedtls/issues/8759).

This is due to the usage of mbedtls_x509_crt_parse_file, and the check to make sure this is non-zero within libavformat/tls_mbedtls.c. This should check if the return code is < 0, with values greater than 0 deemed acceptable. A value greater than 0 overall represents a success, just that certain certificates from the file were skipped.

How to reproduce:

Create a ca bundle with various system trust certificates + a certificate with a name constraint. I have attached an example.

% ffplay -i https://google.com -tls_verify 1 -cafile /tmp/ca-bundle.crt
...
mbedtls_x509_crt_parse_file for CA cert returned 1
...

Duplicating the name constrained certificate in the CA file changes and re-running the command changes the return code to 2.

Remove the name constrained certificate from the CA file (its the first one marked as such) and re-run the command. This time it will give an invalid data error and no longer an SSL error. (this is expected as we passed in google.com not a playable URL, and is not an issue being raised)

$ ffplay -version
ffplay version n7.0.1 Copyright (c) 2003-2024 the FFmpeg developers
built with gcc 14.1.1 (GCC) 20240522
configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-frei0r --enable-gmp --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libdav1d --enable-libdrm --enable-libdvdnav --enable-libdvdread --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libharfbuzz --enable-libiec61883 --enable-libjack --enable-libjxl --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libplacebo --enable-libpulse --enable-librav1e --enable-librsvg --enable-librubberband --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpl --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-libzimg --enable-mbedtls --enable-nvdec --enable-nvenc --enable-opencl --enable-opengl --enable-shared --enable-vapoursynth --enable-version3 --enable-vulkan
libavutil      59.  8.100 / 59.  8.100
libavcodec     61.  3.100 / 61.  3.100
libavformat    61.  1.100 / 61.  1.100
libavdevice    61.  1.100 / 61.  1.100
libavfilter    10.  1.100 / 10.  1.100
libswscale      8.  1.100 /  8.  1.100
libswresample   5.  1.100 /  5.  1.100
libpostproc    58.  1.100 / 58.  1.100

[Skybound]

Example CA file