Libav and Free Software development

Tracking down a bug in Clang: -fsanitize=undefined is not bug-free

Leave a comment

In search of a better backtrace, I tried compiling Libav with -fsanitize=undefined and –disable-optimizations, and filed Libav bug 683 when it failed. The failure did not seem reasonable given the source code, and on further examination it looked like the key problem was actually in the C compiler, Clang, rather than the code, so I looked into it and filed a Clang bug on Here is the confirmed Clang bug report.

No one wants to deal with 4000 lines of preprocessed source code if there is a better alternative, so I wrote a creduce script to find what some fairly minimal source code to trigger the bug would look like. It turns out that the problem can be reproduced in a rather tiny way: fn1() { __asm__(“” ::”r”(0), “i”(0 * 0)); }

This is a one-line file of C source code which uses inline assembly.

An acquaintance of mine asked for a detailed writeup, so here it is, in full detail, from narrowing down the problematic compiler flag combination onwards. If you are curious to see how to narrow down a bug like this by example, read on; otherwise, please skip the rest of this post.

First, I ran ../configure --toolchain=clang-usan --disable-optimizations && make -j8 This led to a compiler error, so I tried make V=1 -j8 to get more details; if I’d started with that, I would have run:
../configure --toolchain=clang-usan --disable-optimizations && make V=1 -j8

There was a lot of output, but here is the relevant part:

clang -I. -Iadir/libav -D_ISOC99_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_POSIX_C_SOURCE=200112 -D_XOPEN_SOURCE=600 -DHAVE_AV_CONFIG_H -fsanitize=undefined   -std=c99 -pthread -g -Wdeclaration-after-statement -Wall -Wdisabled-optimization -Wpointer-arith -Wredundant-decls -Wcast-qual -Wwrite-strings -Wtype-limits -Wundef -Wmissing-prototypes -Wstrict-prototypes -Wno-parentheses -Wno-switch -Wno-format-zero-length -Wno-pointer-sign  -fno-math-errno -fno-signed-zeros -Qunused-arguments -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=return-type  -MMD -MF libavcodec/x86/dsputil_x86.d -MT libavcodec/x86/dsputil_x86.o -c -o libavcodec/x86/dsputil_x86.o adir/libav/libavcodec/x86/dsputil_x86.c

adir/libav/libavcodec/x86/dsputil_mmx.c:186:1: error: invalid operand for inline asm constraint ‘i’
CLEAR_BLOCKS(ff_clear_blocks_mmx, 6)
adir/libav/libavcodec/x86/dsputil_mmx.c:173:9: note: expanded from macro ‘CLEAR_BLOCKS’
“pxor %%mm7, %%mm7              \n\t”           \
adir/libav/libavcodec/x86/dsputil_mmx.c:187:1: error: invalid operand for inline asm constraint ‘i’
CLEAR_BLOCKS(ff_clear_block_mmx, 1)
adir/libav/libavcodec/x86/dsputil_mmx.c:173:9: note: expanded from macro ‘CLEAR_BLOCKS’
“pxor %%mm7, %%mm7              \n\t”           \
/home/me/opw/libav/libavcodec/x86/dsputil_mmx.c:208:9: error: invalid operand for inline asm constraint ‘i’
“xorps  %%xmm0, %%xmm0              \n”

3 errors generated.

So, aside from telling me that there is an error, I have the command that triggered it:

clang -I. -I/media/spinny/me/opw/libav -D_ISOC99_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_POSIX_C_SOURCE=200112 -D_XOPEN_SOURCE=600 -DHAVE_AV_CONFIG_H -fsanitize=undefined   -std=c99 -pthread -g -Wdeclaration-after-statement -Wall -Wdisabled-optimization -Wpointer-arith -Wredundant-decls -Wcast-qual -Wwrite-strings -Wtype-limits -Wundef -Wmissing-prototypes -Wstrict-prototypes -Wno-parentheses -Wno-switch -Wno-format-zero-length -Wno-pointer-sign  -fno-math-errno -fno-signed-zeros -Qunused-arguments -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=return-type  -MMD -MF libavcodec/x86/dsputil_mmx.d -MT libavcodec/x86/dsputil_mmx.o -c -o libavcodec/x86/dsputil_mmx.o /media/spinny/me/opw/libav/libavcodec/x86/dsputil_mmx.c

So, the next step is to get the preprocessed source, and reproduce the bug with that. First, preprocess the source:

clang -I. -I/media/spinny/me/opw/libav -D_ISOC99_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_POSIX_C_SOURCE=200112 -D_XOPEN_SOURCE=600 -DHAVE_AV_CONFIG_H -fsanitize=undefined   -std=c99 -pthread -g -Wdeclaration-after-statement -Wall -Wdisabled-optimization -Wpointer-arith -Wredundant-decls -Wcast-qual -Wwrite-strings -Wtype-limits -Wundef -Wmissing-prototypes -Wstrict-prototypes -Wno-parentheses -Wno-switch -Wno-format-zero-length -Wno-pointer-sign  -fno-math-errno -fno-signed-zeros -Qunused-arguments -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=return-type  -MMD -MF libavcodec/x86/dsputil_mmx.d -MT libavcodec/x86/dsputil_mmx.o -E -o libavcodec/x86/dsputil_mmx.i /media/spinny/me/opw/libav/libavcodec/x86/dsputil_mmx.c

Then, make sure the preprocessed source still does the same thing – it should, but it is a way to double-check for obvious typos or errors before the next step:
clang -I. -I/media/spinny/me/opw/libav -D_ISOC99_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_POSIX_C_SOURCE=200112 -D_XOPEN_SOURCE=600 -DHAVE_AV_CONFIG_H -fsanitize=undefined   -std=c99 -pthread -g -Wdeclaration-after-statement -Wall -Wdisabled-optimization -Wpointer-arith -Wredundant-decls -Wcast-qual -Wwrite-strings -Wtype-limits -Wundef -Wmissing-prototypes -Wstrict-prototypes -Wno-parentheses -Wno-switch -Wno-format-zero-length -Wno-pointer-sign  -fno-math-errno -fno-signed-zeros -Qunused-arguments -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=return-type  -MMD -MF libavcodec/x86/dsputil_mmx.d -MT libavcodec/x86/dsputil_mmx.o -c -o libavcodec/x86/dsputil_mmx.o /media/spinny/me/opw/libav/build_vubsan/libavcodec/x86/dsputil_mmx.i

It still gives the same error, as it should. It is important to end the filename with .i; Clang is quite picky about file extensions, and will do something else entirely if you give it a different extension. Now, whittle down arguments to the minimum that can reproduce the problem. It turned out that running -fsanitize=undefined by itself on the preprocessed source was enough (the -c -o something.o says to compile and output to an object file, without linking – it is necessary, because the point is to take away more and more until only the problem is left, so requiring a big subset of the project to link against would be bad). The following command causes the same set of three errors shown above.
clang  -fsanitize=undefined     -c -o libavcodec/x86/dsputil_mmx.o adir/libav/build_vubsan/libavcodec/x86/dsputil_mmx.i

Now that the minimal command line arguments have been found, it is time to try to make the code that triggers the problem smaller. Install creduce (sudo aptitude install creduce), and write a creduce test – it is a shell script, which I saved in a file called

rm -f out.txt *.o
ulimit -t 1

clang -c -o good.o dsputil_mmx.i &&\
! clang -fsanitize=undefined -c -o no.o dsputil_mmx.i > out.txt 2>&1 &&\
grep “invalid operand for inline asm constraint” out.txt
then exit 0
else exit 1

This test has a few parts. Commented with #, here they are:
#!/bin/bash # Specify that the test script should be interpreted with the bash shell.
rm -f out.txt *.o # Clean up from previous runs; creduce will run the script a lot of times.
ulimit -t 1 # Limit the process time to one second, so creduce cannot cause infinite loops to stop progress on finding smaller code.

Everything else is wrapped in the following structure:
if (a bunch of stuff) then success, else failure, fi (fi ends the block started by the if).

That just leaves the core of the test to explain: it can be an arbitrary number of conditions, but for this, three were enough.
clang -c -o good.o dsputil_mmx.i # This says that the reduced code should be able to compile with Clang. creduce does not guarantee that it produces valid source code, but this step limits it to plausible source that Clang is willing to compile.

! clang -fsanitize=undefined -c -o no.o dsputil_mmx.i # The ! is negation: it should fail to compile with -fsanitize=undefined, indicated by having a non-zero return code. The difference of the name of the object file, no.o, is cosmetic, not important.

(the above command) > out.txt 2>&1 # Redirect both normal and error output of the command into the same file descriptor, and put it in a file called out.txt.

grep “invalid operand for inline asm constraint” out.txt # Make sure the error that the original source code had is still in the output the compiler gives, rather than potentially being an entirely different error.

The && bits chain everything together, and the \ at the end of the lines let it be formatted readably; having everything on one line would work, but not be nice.

Then, run creduce on the preprocessed source from before (after copying it to the directory with in it):
creduce dsputil_mmx.i

Look at the minimal file created, , check it for plausibility, manually compile it if you feel particularly cautious – it causes the output “dsputil_mmx.i:1:17: error: invalid operand for inline asm constraint ‘i'” when compiled with -fsanitize=undefined, and only has some minor warnings when compiled without it. A slightly more elaborate script could have generated code without the warnings (by having ! grep lines for warnings to avoid), but it does not significantly change anything in this example.

The next step was reporting the bug. I uploaded the original preprocessed source, put the one-line minimal program into the body of the bug report itself, explained what happens with and without -fsanitize=undefined, and showed the creduce script, Clang version I was using, and links to related bugs/patches (the original Libav bug report, and the FFmpeg workaround; this was before the Libav workaround).

At this point, the bug is confirmed, and there is discussion about how to fix it in the bug report.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s