CTest
CTest is the test runner that comes with CMake. A test is a command with a name, and its result is the exit status: zero passes, anything else fails. No framework is needed, and lesson 085's harness is already the right shape.
The CMake side is two lines: enable_testing() to switch the runner on, and add_test() per case. This lesson registers one test, then a failing one, then a test with arguments, and runs them all with ctest.
CTest also has a timeout, a repeat mode and a label system, and a test can be marked as WILL_FAIL or SKIP_RETURN_CODE. A suite that grows uses them instead of wrapping the program in a script.
A suite that runs in a second is run on every change; a suite that takes ten minutes is run before a release. CTest keeps the small ones cheap, which is why the program under test is the test.
A test that needs the program to fail is written with WILL_FAIL, and a test that is not ready is disabled rather than deleted. The runner keeps the suite honest about what it checks.
The tests are part of the build description, so a new developer runs the suite with one command and no documentation. That is worth more than the framework features a bigger tool would add.
enable_testing and add_test
enable_testing() writes the files ctest needs, and add_test(NAME adds COMMAND check) registers the executable as a test. The program returns 0 when its checks pass and 1 when one fails.
The test program prints nothing on success and returns 0; ctest runs it and reports the pass. The output line has the test number, the name and the time it took.
The test runs from the build directory by default, which matters for a test that reads a file. The working directory can be set per test with WORKING_DIRECTORY.
check.cpp
#include <iostream>
int add(int a, int b)
{
return a + b;
}
int main()
{
if (add(2, 3) != 5) {
std::cout << "add is broken\n";
return 1;
}
return 0;
}
CMakeLists.txt
cmake_minimum_required(VERSION 3.20)
project(toolbox CXX)
enable_testing()
add_executable(check check.cpp)
add_test(NAME adds COMMAND check)
cmake -S . -B build
cmake --build build
ctest --test-dir build --output-on-failure
-- The CXX compiler identification is GNU 15.2.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done (0.4s)
-- Generating done (0.0s)
-- Build files have been written to: /tmp/cppcourse/work/L101/s0/build
[ 50%] Building CXX object CMakeFiles/check.dir/check.cpp.o
[100%] Linking CXX executable check
[100%] Built target check
Test project /tmp/cppcourse/work/L101/s0/build
Start 1: adds
1/1 Test #1: adds ............................. Passed 0.01 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 0.06 sec
A failing test
The same project with a wrong expectation returns 1. ctest reports the test as Failed and prints the program's output only with --output-on-failure, which is why the flag belongs in every run.
The program returns 1 from its check, and the summary counts one test and one failure. The exit status of ctest itself is non-zero, so a build server stops.
The test's output is the message from lesson 085, and the CTest report wraps it with the name and the time. That is the whole value of the runner: a name, a status and the output in one place.
check.cpp
#include <iostream>
int add(int a, int b)
{
return a + b;
}
int main()
{
if (add(2, 2) != 5) {
std::cout << "add(2, 2) is " << add(2, 2) << '\n';
return 1;
}
return 0;
}
CMakeLists.txt
cmake_minimum_required(VERSION 3.20)
project(toolbox CXX)
enable_testing()
add_executable(check check.cpp)
add_test(NAME adds COMMAND check)
cmake -S . -B build
cmake --build build
ctest --test-dir build --output-on-failure
-- The CXX compiler identification is GNU 15.2.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done (0.6s)
-- Generating done (0.0s)
-- Build files have been written to: /tmp/cppcourse/work/L101/s1/build
[ 50%] Building CXX object CMakeFiles/check.dir/check.cpp.o
[100%] Linking CXX executable check
[100%] Built target check
Test project /tmp/cppcourse/work/L101/s1/build
Start 1: adds
1/1 Test #1: adds .............................***Failed 0.00 sec
add(2, 2) is 4
0% tests passed, 1 tests failed out of 1
Total Test time (real) = 0.00 sec
The following tests FAILED:
1 - adds (Failed)
Errors while running CTest
Arguments to a test
add_test takes a command and its arguments after the name, so one executable can serve several cases. The program below adds the two numbers from argv and returns 0 when the sum matches the third argument.
The project registers two tests with different arguments, and ctest runs both. The output names each one, and a failure in one does not stop the other.
The pattern scales: a test binary with a mode argument covers a family of cases, and a script can be registered as a test too. ctest only needs a command and an exit status.
check.cpp
#include <cstdlib>
int add(int a, int b)
{
return a + b;
}
int main(int argc, char** argv)
{
if (argc < 4) {
return 2;
}
int a = std::atoi(argv[1]);
int b = std::atoi(argv[2]);
int expected = std::atoi(argv[3]);
return add(a, b) == expected ? 0 : 1;
}
CMakeLists.txt
cmake_minimum_required(VERSION 3.20)
project(toolbox CXX)
enable_testing()
add_executable(check check.cpp)
add_test(NAME two_and_three COMMAND check 2 3 5)
add_test(NAME minus_one COMMAND check -1 1 0)
cmake -S . -B build
cmake --build build
ctest --test-dir build
-- The CXX compiler identification is GNU 15.2.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done (0.4s)
-- Generating done (0.0s)
-- Build files have been written to: /tmp/cppcourse/work/L101/s2/build
[ 50%] Building CXX object CMakeFiles/check.dir/check.cpp.o
[100%] Linking CXX executable check
[100%] Built target check
Test project /tmp/cppcourse/work/L101/s2/build
Start 1: two_and_three
1/2 Test #1: two_and_three .................... Passed 0.00 sec
Start 2: minus_one
2/2 Test #2: minus_one ........................ Passed 0.00 sec
100% tests passed, 0 tests failed out of 2
Total Test time (real) = 0.00 sec
Running part of the suite
ctest -R runs the tests whose name matches the regular expression, and ctest -E runs all but the matches. The program's two tests are run with a filter that picks one.
The output shows one test and the summary counts one, which is how a developer runs the case they are working on without waiting for the rest.
The test names are the interface: a prefix per area, such as parse_ and io_, makes the filter useful. A suite whose names are all test1, test2 is a suite nobody can run in parts.
check.cpp
#include <cstdlib>
int add(int a, int b)
{
return a + b;
}
int main(int argc, char** argv)
{
if (argc < 4) {
return 2;
}
int a = std::atoi(argv[1]);
int b = std::atoi(argv[2]);
int expected = std::atoi(argv[3]);
return add(a, b) == expected ? 0 : 1;
}
CMakeLists.txt
cmake_minimum_required(VERSION 3.20)
project(toolbox CXX)
enable_testing()
add_executable(check check.cpp)
add_test(NAME two_and_three COMMAND check 2 3 5)
add_test(NAME minus_one COMMAND check -1 1 0)
cmake -S . -B build
cmake --build build
ctest --test-dir build -R two
-- The CXX compiler identification is GNU 15.2.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done (0.4s)
-- Generating done (0.0s)
-- Build files have been written to: /tmp/cppcourse/work/L101/s3/build
[ 50%] Building CXX object CMakeFiles/check.dir/check.cpp.o
[100%] Linking CXX executable check
[100%] Built target check
Test project /tmp/cppcourse/work/L101/s3/build
Start 1: two_and_three
1/1 Test #1: two_and_three .................... Passed 0.00 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 0.00 sec
Exercises
- Add a test that expects the program to fail and check that ctest reports it as a failure.
- Register a test that runs a shell script and returns its exit status.
- Run ctest with -j 4 and compare the output with a serial run.
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
What decides whether a CTest test passed?
Answer
The exit status of the command
CTest runs the command and reads its status: zero is a pass, anything else a failure.
-
Which CMake command switches the test runner on?
Answer
enable_testing()enable_testing writes the test files; add_test then registers each case.
-
What does this CMake line register?
add_test(NAME adds COMMAND check)Answer
A test called adds that runs the executable check
NAME is the test's name and COMMAND is what runs; the exit status decides the result.
-
What does ctest print for a passing test by default?
Answer
A line with the test number, the name and the time
The test output is hidden unless it fails or --output-on-failure is given.
-
What is the exit status of ctest when one test fails?
Answer
1
A non-zero status stops a build server, which is the point of running the suite there.
-
What does
ctest -R twodo?Answer
Runs the tests whose names match the expression two
-R takes a regular expression over the test names.
-
Where does a test run by ctest execute by default?
Answer
In the build directory
The build tree is the working directory unless WORKING_DIRECTORY says otherwise.
-
What does --output-on-failure add?
Answer
The output of a test only when it fails
A passing test stays quiet; a failing one shows what it printed.
-
What does a test program return to signal success?
Answer
0
Zero is success, as for every program; main returning a count of failures must return 0 when there are none.
-
Why is a name prefix such as parse_ useful for tests?
Answer
It makes -R and -E filters select a group of tests
The filter matches names, so a prefix turns the suite into groups that can be run alone.
The code on this page was built with g++ 15.2.0 and CMake 4.2.3 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container. The programs whose section shows a compiler error were compiled until that error and were not run.
Article author: Arthur Isaev