dotnet test with xUnit

xUnit is a test framework: a package of assertions, a test SDK that discovers the tests in an assembly, and a runner that dotnet test starts. The SDK template writes the four files a test project needs, and the whole suite is then one command whose status a build server can read. This page runs the template against the calculator library of the previous page.

The template

dotnet new xunit -o tests writes the project, dotnet new sln writes a solution and dotnet sln add puts both projects in it. A solution is what makes a bare dotnet test in the repository root find the tests; without one it stops with MSBUILD : error MSB1003, which the page shows below.

The project file of the test project names the four packages of the template, and every one of them is a NuGet package:

<Project Sdk="Microsoft.NET.Sdk"> <PropertyGroup> <TargetFramework>net10.0</TargetFramework> <ImplicitUsings>enable</ImplicitUsings> <Nullable>enable</Nullable> <IsPackable>false</IsPackable> </PropertyGroup> <ItemGroup> <PackageReference Include="coverlet.collector" Version="6.0.4" /> <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.14.1" /> <PackageReference Include="xunit" Version="2.9.3" /> <PackageReference Include="xunit.runner.visualstudio" Version="3.1.4" /> </ItemGroup> <ItemGroup> <Using Include="Xunit" /> </ItemGroup> <ItemGroup> <ProjectReference Include="..\src\Calc\Calc.csproj" /> </ItemGroup> </Project>

The test file keeps the class of the library under test in its own file. A [Fact] is one case, and a [Theory] with [InlineData] is one case per row:

using Calc; namespace Tests; public class CalculatorTests { [Fact] public void Add_adds_two_numbers() { Assert.Equal(5, Calculator.Add(2, 3)); } [Theory] [InlineData(10, 2, 5)] [InlineData(9, 3, 3)] [InlineData(7, 7, 1)] public void Divide_divides(int left, int right, int expected) { Assert.Equal(expected, Calculator.Divide(left, right)); } [Fact] public void Divide_by_zero_throws() { Assert.Throws<DivideByZeroException>(() => Calculator.Divide(1, 0)); } }

The file has no Main, so it is not a program that runs on its own: the test SDK generates the entry point that finds the methods, and that entry point is what the assembly is run with. The using Calc; line works because of the ProjectReference of the previous page.

A passing run

dotnet test restores, builds and runs. The last line is the one that matters: the counts and the status of the process:

dotnet test
echo "status: $?"

  Determining projects to restore...
  Restored /tmp/r10/xt/tests/tests.csproj (in 739 ms).
  1 of 2 projects are up-to-date for restore.
  Calc -> /tmp/r10/xt/src/Calc/bin/Debug/net10.0/Calc.dll
  tests -> /tmp/r10/xt/tests/bin/Debug/net10.0/tests.dll
Test run for /tmp/r10/xt/tests/bin/Debug/net10.0/tests.dll (.NETCoreApp,Version=v10.0)
A total of 1 test files matched the specified pattern.

Passed!  - Failed:     0, Passed:     5, Skipped:     0, Total:     5, Duration: 64 ms - tests.dll (net10.0)
status: 0

Five tests and not three: the theory with three rows counts as three cases. The duration is measured by the runner and moves between runs; the counts do not. The paths in the two build lines are the folders of the projects on the machine that ran it, which is this container.

A failing test

The same suite with one expectation changed to six. The runner prints the failed test, the message of the assertion, the stack trace and a summary, and the status is one:

dotnet test
echo "status: $?"

  Determining projects to restore...
  All projects are up-to-date for restore.
  Calc -> /tmp/r10/xt/src/Calc/bin/Debug/net10.0/Calc.dll
  tests -> /tmp/r10/xt/tests/bin/Debug/net10.0/tests.dll
Test run for /tmp/r10/xt/tests/bin/Debug/net10.0/tests.dll (.NETCoreApp,Version=v10.0)
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:01.25]     Tests.CalculatorTests.Add_adds_two_numbers [FAIL]
  Failed Tests.CalculatorTests.Add_adds_two_numbers [64 ms]
  Error Message:
   Assert.Equal() Failure: Values differ
Expected: 6
Actual:   5
  Stack Trace:
     at Tests.CalculatorTests.Add_adds_two_numbers() in /tmp/r10/xt/tests/CalculatorTests.cs:line 10
   at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

Failed!  - Failed:     1, Passed:     4, Skipped:     0, Total:     5, Duration: 506 ms - tests.dll (net10.0)
status: 1

The two lines Expected: 6 and Actual: 5 are the whole value of an assertion library: a reader does not have to add a print to see what the value was. The stack trace names the file and the line, and the path is the build folder of the machine that ran the test.

A filter

A filter runs a part of the suite. FullyQualifiedName~Divide selects the tests whose full name contains the word, which is the two divide tests and their three theory rows:

dotnet test --filter "FullyQualifiedName~Divide"
echo "status: $?"

  Determining projects to restore...
  All projects are up-to-date for restore.
  Calc -> /tmp/r10/xt/src/Calc/bin/Debug/net10.0/Calc.dll
  tests -> /tmp/r10/xt/tests/bin/Debug/net10.0/tests.dll
Test run for /tmp/r10/xt/tests/bin/Debug/net10.0/tests.dll (.NETCoreApp,Version=v10.0)
A total of 1 test files matched the specified pattern.

Passed!  - Failed:     0, Passed:     4, Skipped:     0, Total:     4, Duration: 262 ms - tests.dll (net10.0)
status: 0

Four tests and not five, because the addition test does not match the filter. A filter is what a developer runs while working on one class, and the full suite is what a build server runs before a merge.

A restore without a feed

The four packages have to come from somewhere. The image of this container carries the packages of the xUnit template and coverlet in its NuGet cache, so a fresh project restores them without the network. A project whose packages are not in the cache is a different matter: the restore asks nuget.org, and with no route to the feed it stops with NU1301. The command below removes the route for the length of one command with a proxy that refuses every connection, which is the state of a machine that has no network:

https_proxy=http://127.0.0.1:9 http_proxy=http://127.0.0.1:9 NUGET_PACKAGES=/tmp/empty dotnet test -p:RestoreNoCache=true
echo "status: $?"

/tmp/r10/XunitOffline/XunitOffline.csproj : error NU1301:   Connection refused
/tmp/r10/XunitOffline/XunitOffline.csproj : error NU1301: Failed to retrieve information about 'xunit.runner.visualstudio' from remote source 'https://api.nuget.org/v3-flatcontainer/xunit.runner.visualstudio/index.json'.
/tmp/r10/XunitOffline/XunitOffline.csproj : error NU1301:   Connection refused (127.0.0.1:9)
/tmp/r10/XunitOffline/XunitOffline.csproj : error NU1301:   Connection refused
/tmp/r10/XunitOffline/XunitOffline.csproj : error NU1301: Failed to retrieve information about 'Microsoft.NET.Test.Sdk' from remote source 'https://api.nuget.org/v3-flatcontainer/microsoft.net.test.sdk/index.json'.
/tmp/r10/XunitOffline/XunitOffline.csproj : error NU1301:   Connection refused (127.0.0.1:9)
/tmp/r10/XunitOffline/XunitOffline.csproj : error NU1301:   Connection refused
/tmp/r10/XunitOffline/XunitOffline.csproj : error NU1301: Failed to retrieve information about 'coverlet.collector' from remote source 'https://api.nuget.org/v3-flatcontainer/coverlet.collector/index.json'.
/tmp/r10/XunitOffline/XunitOffline.csproj : error NU1301:   Connection refused (127.0.0.1:9)
/tmp/r10/XunitOffline/XunitOffline.csproj : error NU1301:   Connection refused
  Failed to restore /tmp/r10/XunitOffline/XunitOffline.csproj (in 6.14 sec).
status: 1

The output is the last lines of a longer report: each of the four packages was asked for several times before NuGet gave up, and the retries say Connection refused (127.0.0.1:9) because that is the proxy of this measurement. The code is the same with no proxy at all: NU1301, the feed could not be read. The status is one, so a build server treats it as a failed build and not as a failed test.

The next page runs the same suite with the coverage collector of the template and reads the file it writes.

Exercises

  1. Add a fact that asserts the exception message of the division by zero.
  2. Run the suite with --filter "FullyQualifiedName~Add" and check the count.
  3. Run dotnet test in the folder above the solution and explain the error.
  4. Add a theory row that fails and read the line the runner prints for it.

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. What does [Fact] mark?

    Answer

    One test method with no parameters

    A theory with InlineData is the parameterised form.

  2. How many test cases does the theory of the page add?

    Answer

    Three, one per InlineData row

    One fact plus three rows plus one fact is the five of the run.

  3. What does this program print? var values = new[] { 10, 2, 5 }; Console.WriteLine(values[0] / values[1]);

    Answer

    5

    Ten divided by two is the first row of the theory.

  4. What is the status of a run with one failed test?

    Answer

    1

    The measured failing run ends with status 1 and a Failed! line.

  5. What does this program print? Console.WriteLine($"{5} of {5}");

    Answer

    5 of 5

    Both holes are filled with the value five.

  6. Why does dotnet test in a folder without a project stop with MSB1003?

    Answer

    It needs a project or a solution to know what to run

    The message says to specify a project or a solution file.

  7. What does this program print? var name = "Tests.CalculatorTests.Divide_divides"; Console.WriteLine(name.Contains("Divide"));

    Answer

    True

    The filter of the page matches on the same kind of name.

  8. What does NU1301 report?

    Answer

    That a package could not be read from its remote source

    The measured line names the source and the connection that was refused.

  9. What does this program print? var counts = new[] { 0, 0, 5, 0 }; Console.WriteLine(counts[2]);

    Answer

    5

    The third element is the number of passed tests of the green run.

  10. Why is a solution useful here?

    Answer

    A bare dotnet test in the root finds the test project through it

    The page shows the MSB1003 error of a folder without one.

The code on this page was built with the .NET SDK 10.0.303 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev