Coverage with coverlet
Coverage answers one question: which lines did the tests execute? The answer is a number per line, and the number is useful and easy to misread. This page runs the coverage collector of the xUnit template, reads the file it writes with a program of its own, and shows three test states in which the line percentage stays at a hundred while the branch percentage does not.
The collector
The test project of the previous page already references coverlet.collector, so the collector is one option of dotnet test and not an install. The file it writes is attached to the run, and the last lines of the output name it:
dotnet test --collect:"XPlat Code Coverage"
echo "status: $?"
Determining projects to restore...
All projects are up-to-date for restore.
Calc -> /tmp/r10/xt/src/Calc/bin/Debug/net10.0/Calc.dll
tests -> /tmp/r10/xt/tests/bin/Debug/net10.0/tests.dll
Test run for /tmp/r10/xt/tests/bin/Debug/net10.0/tests.dll (.NETCoreApp,Version=v10.0)
A total of 1 test files matched the specified pattern.
Passed! - Failed: 0, Passed: 5, Skipped: 0, Total: 5, Duration: 101 ms - tests.dll (net10.0)
Attachments:
/tmp/r10/xt/tests/TestResults/71f7d774-b599-431c-81ad-5bc9b371ee4a/coverage.cobertura.xml
The name of the folder under TestResults is a fresh identifier per run, so a script finds the file with find rather than by writing the path. The format is cobertura XML, which the other .NET tools and the build servers read.
The cobertura file
The root element of the file holds the summary of the whole run, and one attribute of it is the number people quote:
<?xml version="1.0" encoding="utf-8"?>
<coverage line-rate="1" branch-rate="1" version="1.9" timestamp="1795258337" lines-covered="4" lines-valid="4" branches-covered="2" branches-valid="2">
<sources>
- line-rate is the fraction of the instrumented lines that ran, from zero to one, and lines-covered and lines-valid are the two numbers behind it.
- branch-rate is the fraction of the branches taken: the two sides of an if, the two arms of a ternary, each case of a switch.
- version="1.9" is the version of the coverage tool that wrote the file, not of the code and not of the runtime.
A reader for it
The file is XML, and the program below turns it into five lines by reading four attributes. It is a console project of its own; the only part worth knowing is XDocument, which is System.Xml.Linq from the base class library:
using System.Xml.Linq;
var path = args.Length > 0 ? args[0] : "coverage.cobertura.xml";
if (!File.Exists(path))
{
Console.WriteLine("usage: coverage <path of coverage.cobertura.xml>");
return 0;
}
var root = XDocument.Load(path).Root!;
var lines = (int)root.Attribute("lines-valid")!;
var covered = (int)root.Attribute("lines-covered")!;
var branches = (int)root.Attribute("branches-valid")!;
var branchCovered = (int)root.Attribute("branches-covered")!;
Console.WriteLine($"file {Path.GetFileName(path)}");
Console.WriteLine($"lines {covered} of {lines}");
Console.WriteLine($"branches {branchCovered} of {branches}");
Console.WriteLine($"line rate {(double)covered / lines:P0}");
Console.WriteLine($"branch rate {(double)branchCovered / Math.Max(branches, 1):P0}");
return 0;
coverage=$(find tests/TestResults -name coverage.cobertura.xml | head -1)
dotnet run -- "$coverage"
echo "status: $?"
file coverage.cobertura.xml
lines 4 of 4
branches 2 of 2
line rate 100 %
branch rate 100 %
status: 0
The percentage is formatted with :P0, and the invariant culture of this container writes it with a space before the sign, which is why the output reads 100 % and not 100%. The division guards the branch count with Math.Max(branches, 1), because a method without a branch has zero valid branches and the rate would be a division by zero. The first check prints a usage line when the file is not there, so the program is also readable when it is started from the wrong folder:
dotnet run
echo "status: $?"
usage: coverage <path of coverage.cobertura.xml>
status: 0
Three states
The library has two methods: Add is one line and Divide is a ternary with two branches. Three versions of the test class, each run with the same collector and the same reader:
| Tests | Lines | Branches | Line rate | Branch rate |
|---|---|---|---|---|
| the addition test only | 1 of 4 | 0 of 2 | 25 % | 0 % |
| addition and the exception | 4 of 4 | 1 of 2 | 100 % | 50 % |
| all five tests | 4 of 4 | 2 of 2 | 100 % | 100 % |
The middle row is the lesson. Every line ran, so a reader who quotes the line rate sees a fully covered library; the true half of the ternary, the division that returns a value, never ran, and only the branch rate says so. A test of the exception proves the error path and says nothing about the path that returns.
The first row is the other direction: a single test that covers a quarter of the lines and none of the branches. A percentage without the two counts behind it cannot be compared between projects, because a library of one method and a library of a thousand reach the same number from very different places.
A green status with no tests
The console test project of the previous page is a program for dotnet run and invisible to dotnet test. Running the test command on it does not fail: it restores, builds nothing that it can run, and reports success:
dotnet test tests/Calc.Tests/Calc.Tests.csproj
echo "status: $?"
Determining projects to restore...
All projects are up-to-date for restore.
status: 0
A status of zero with no tests is worse than a red run, because a build server and a person both read it as a passing suite. The test SDK is what makes a project a test project: it adds the entry point that lists the tests in the assembly, and a project without it has no tests to list.
- dotnet test on a project with the test SDK and no tests prints No test is available and still returns zero, which is another green result to read with care.
- A suite that must not be empty needs a test that asserts the count of the discovered tests, or a build step that fails when the count is zero.
- The collector of this page is a package of the test project, so a console project has neither tests nor coverage.
The last page of the group is the reference for the frameworks and libraries around these tests: NUnit, MSTest, Moq, FluentAssertions and BenchmarkDotNet, each installed and run here, with the version that was restored.
Exercises
- Add a test that covers the true branch of the division and check the two rates of the middle row.
- Delete the exception test and describe the row the table would show.
- Print the per-class rates of the cobertura file, which are nested one level below the root.
- Run the collector on a project with no tests and explain the file it does or does not write.
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
What does coverlet.collector write?
Answer
A cobertura XML file under TestResults
The measured run names the coverage.cobertura.xml it attached.
-
What does a branch rate of 50 % mean?
Answer
Half of the branches were taken, for example one side of a ternary
The middle row of the table has every line and one of two branches.
-
What does this program print?
var covered = 1; var valid = 4; Console.WriteLine($"{(double)covered / valid:P0}");Answer
25 %
The invariant culture of the container writes a space before the sign.
-
Why is the line rate 100 % in the middle row while a branch is not taken?
Answer
Every line ran, and one branch of the ternary did not
The page names this as the lesson of the middle row.
-
What does this program print?
var lines = 4; var covered = 4; Console.WriteLine($"{covered} of {lines}");Answer
4 of 4
That is the line count of the last two rows of the table.
-
What does dotnet test do on a console project without the test SDK?
Answer
It restores, finds no tests and returns zero
The measured run printed the restore lines and status 0.
-
Why is a status of zero without tests dangerous?
Answer
It looks like a passing suite to a build server and to a person
The page calls it worse than a red run.
-
What does this program print?
var branches = 0; Console.WriteLine($"{(double)0 / Math.Max(branches, 1):P0}");Answer
0 %
The guard keeps the division out of zero.
-
Which attribute holds the number of lines that could run?
Answer
lines-valid
The covered count is the other attribute of the pair.
-
Which test state has the best branch rate?
Answer
The five tests, with 2 of 2 branches
The last row of the table is the only one at 100 % on both rates.
The code on this page was built with the .NET SDK 10.0.303 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev