The exit status is the result
A test suite is a program that other programs run, and the only thing they read is the exit status. A green run is zero and a red run is not, and everything else, from the message on the screen to the file of results, is for the person who reads the log. This page follows the status from the driver through the script that launches it to the shell, and measures all four combinations.
The driver
The driver has a passing mode and a failing one, chosen by an argument, so one program produces both statuses. The third assertion takes its expected value from the argument, which is a small and honest way to make a suite fail on demand:
Console.WriteLine("driver 1.0");
var broken = args.Contains("--break");
var check = new Check();
check.Equal(4, 2 + 2, "two and two");
check.Equal(9, 3 * 3, "three threes");
check.Equal(broken ? 7 : 6, 1 + 2 + 3, "one two three");
Console.WriteLine($"checks {check.Passed} passed {check.Failed} failed");
return check.Failed == 0 ? 0 : 1;
sealed class Check
{
public int Passed { get; private set; }
public int Failed { get; private set; }
public void Equal<T>(T expected, T actual, string name)
{
if (EqualityComparer<T>.Default.Equals(expected, actual))
{
Passed++;
Console.WriteLine($"ok {name}");
return;
}
Failed++;
Console.WriteLine($"FAIL {name}: expected {expected}, got {actual}");
}
}
A real suite would not have a switch that breaks it, of course. The switch stands for the defect that the suite is supposed to catch, and it lets this page show the two statuses of the same program without a second file.
Two direct runs
The first run passes and the second one passes the argument after --, which is where the arguments of the program begin:
dotnet run
echo "status: $?"
driver 1.0
ok two and two
ok three threes
ok one two three
checks 3 passed 0 failed
status: 0
dotnet run -- --break
echo "status: $?"
driver 1.0
ok two and two
ok three threes
FAIL one two three: expected 7, got 6
checks 2 passed 1 failed
status: 1
dotnet run keeps the status of the program it ran, which is what makes the first line of a script work at all. The status is one for the failing run and not a count of failures: a shell has one byte of status, and a suite with two failures still reports one.
The script
The script is what a build server or a pre-commit hook runs. It has two jobs: run the driver, and keep its status. set -e makes the shell stop at the first command that fails, and the line after the driver is therefore reached only when the driver passed:
#!/bin/sh
# The status of the driver is the status of the suite.
set -e
dotnet run -- "$@"
echo "suite passed"
The same two runs through the script, and the difference is the suite passed line. A failed run stops inside the script, so the line that celebrates is not printed and the status of the script is the status of the driver:
sh run-tests.sh
echo "status: $?"
driver 1.0
ok two and two
ok three threes
ok one two three
checks 3 passed 0 failed
suite passed
status: 0
sh run-tests.sh -- --break
echo "status: $?"
driver 1.0
ok two and two
ok three threes
FAIL one two three: expected 7, got 6
checks 2 passed 1 failed
status: 1
The four statuses
| Run | Program status | Script status | The script prints |
|---|---|---|---|
| driver, no argument | 0 | 0 | three ok lines, the summary and suite passed |
| driver, --break | 1 | 1 | two ok lines, the FAIL line and the summary |
The status of the script is not always the status of the driver: a script that ends with its own echo would report zero after a failed driver, which is the classic way a CI turns red into green. set -e or an explicit exit $? is what keeps the two equal, and the page above shows the first of the two.
- command; echo "status: $?" prints the status of the command, and echo then replaces $? with zero.
- command && next runs the next command only after a success, and command || fallback runs the fallback only after a failure.
- A status above 128 is a signal on Linux: 134 is the status of an unhandled exception, which lesson 062 measured.
The next page moves from the hand-written driver to the framework that ships with the SDK templates: xUnit, the test SDK and a real dotnet test.
Exercises
- Print the number of failures as the status of a program with more than failures and explain what the shell receives.
- Write a second script that runs two drivers and stops at the first non-zero status.
- Make the driver treat an unknown argument as a usage error with status 2.
- Run the script with || echo failed and describe the line the shell adds.
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
What does dotnet run do with the status of the program?
Answer
It returns the same status
The measured failing run ends with status 1.
-
What does set -e do in the script?
Answer
It stops the script at the first command that fails
The failed run through the script never prints suite passed.
-
What does this program print?
Console.WriteLine("checks 2 passed 1 failed"); return 1;Answer
checks 2 passed 1 failed
The return value is the status and is not printed by the program.
-
What is wrong with a script that ends with echo after the driver?
Answer
The status becomes zero, so a failed driver looks green
The page names this as the classic way to turn red into green.
-
What does this program print?
Console.WriteLine("a"); return 3;Answer
a
The return value is the status and the program never prints it.
-
What does command && next do after a failure?
Answer
It skips next
The second command runs only after a success.
-
What does this program print?
var failures = 2; Console.WriteLine(failures); return failures;Answer
2
The number is printed, and the shell keeps only the low byte of it as the status.
-
Why is the status one and not the number of failures?
Answer
A suite reports success or failure, and a log carries the count
The page says two failures still report one.
-
What does this program print with the argument --help?
var words = new[] { "--help" }; Console.WriteLine($"help {words.Contains("--help")}");Answer
help True
Contains finds the argument in the array of the program.
-
What is the status of an unhandled exception on this runtime?
Answer
134
Lesson 062 measured 134, which is 128 plus the signal number.
The code on this page was built with the .NET SDK 10.0.303 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev