The Benchmark library provides a number of functions whose primary purpose in to affect assembly generation, including DoNotOptimize
and ClobberMemory
. In addition there are other functions, such as KeepRunning
, for which generating good assembly is paramount.
For these functions it's important to have tests that verify the correctness and quality of the implementation. This requires testing the code generated by the compiler.
This document describes how the Benchmark library tests compiler output, as well as how to properly write new tests.
Writing a test has two steps:
// CHECK
lines to match against the verified assembly.Example:
LLVM's Filecheck is used to test the generated assembly against the // CHECK
lines specified in the tests source file. Please see the documentation linked above for information on how to write CHECK
directives.
CHECK
directives don't have to match on the exact next line after the previous match, so tests should omit checks for unimportant bits of assembly. (CHECK-NEXT
can be used to ensure a match occurs exactly after the previous match).-O3 -g0
. So we're only testing the optimized output.tools/strip_asm.py
. This removes comments, assembler directives, and unused labels before the test is run.<build-directory>/test/<test-name>.s
CHECK
prefixes to specify lines that should only match in certain situations. The Benchmark tests use CHECK-CLANG
and CHECK-GNU
for lines that are only expected to match Clang or GCC's output respectively. Normal CHECK
lines match against all compilers. (Note: CHECK-NOT
and CHECK-LABEL
are NOT prefixes. They are versions of non-prefixed CHECK
lines)extern "C"
to disable name mangling for specific functions. This makes them easier to name in the CHECK
lines.Writing tests which check the code generated by a compiler are inherently non-portable. Different compilers and even different compiler versions may generate entirely different code. The Benchmark tests must tolerate this.
LLVM Filecheck provides a number of mechanisms to help write "more portable" tests; including matching using regular expressions, allowing the creation of named variables for later matching, and checking non-sequential matches.
For example, say GCC stores a variable in a register but Clang stores it in memory. To write a test that tolerates both cases we "capture" the destination of the store, and then use the captured expression to write the remainder of the test.
Often tests require testing assembly lines which may subtly differ between compilers or compiler versions. A common example of this is matching stack frame addresses. In this case regular expressions can be used to match the differing bits of output. For example:
The tests require Filecheck to be installed along the PATH
of the build machine. Otherwise the tests will be disabled.
Additionally, as mentioned in the previous section, codegen tests are inherently non-portable. Currently the tests are limited to:
Further work could be done, at least on a limited basis, to extend the tests to other architectures and compilers (using CHECK
prefixes).
Furthermore, the tests fail for builds which specify additional flags that modify code generation, including --coverage
or -fsanitize=
.