NUnit, MSTest, Moq, FluentAssertions and BenchmarkDotNet

xUnit is not the only framework, and an assertion library is not the only package a suite uses. This page is the reference for the others: what each one is for, which version was restored, and what it printed when it ran in this container. Every command below was run here, and the image of the container carries the packages of the xUnit template and coverlet in its NuGet cache; the rest were restored from nuget.org for this page, with the restore times quoted as they were measured.

NUnit and MSTest

NUnit and MSTest are the two frameworks older than xUnit. The templates write a test class with their own attributes, and the command is the same dotnet test, because the test SDK is what talks to the runner:

namespace NUnitTests; public class Tests { [SetUp] public void Setup() { } [Test] public void Test1() { Assert.Pass(); } }

namespace MSTestTests; [TestClass] public sealed class Test1 { [TestMethod] public void TestMethod1() { } }

dotnet new nunit and dotnet new mstest write those files, with the packages the templates need. Both ran their one template test here:

  Determining projects to restore...
  Restored /tmp/r10/NUnitTests/NUnitTests.csproj (in 649 ms).
  NUnitTests -> /tmp/r10/NUnitTests/bin/Debug/net10.0/NUnitTests.dll
Test run for /tmp/r10/NUnitTests/bin/Debug/net10.0/NUnitTests.dll (.NETCoreApp,Version=v10.0)
A total of 1 test files matched the specified pattern.

Passed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1, Duration: 53 ms - NUnitTests.dll (net10.0)
status: 0

  Determining projects to restore...
  Restored /tmp/r10/MSTestTests/MSTestTests.csproj (in 21.8 sec).
  MSTestTests -> /tmp/r10/MSTestTests/bin/Debug/net10.0/MSTestTests.dll
Test run for /tmp/r10/MSTestTests/bin/Debug/net10.0/MSTestTests.dll (.NETCoreApp,Version=v10.0)
A total of 1 test files matched the specified pattern.

Passed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1, Duration: 68 ms - MSTestTests.dll (net10.0)
status: 0

The NUnit restore took 649 milliseconds because its packages were already in the NuGet cache of the machine by then; the MSTest restore took 21.8 seconds and downloaded them. That difference is the whole answer to "can a suite run without a network": the first project of a package on a machine cannot, and every project after it can.

Moq and FluentAssertions

Moq makes a stand-in for an interface, and FluentAssertions writes an assertion as a sentence. They are libraries and not frameworks: they are referenced by a test project of any framework, and the page of this group on xUnit shows the run. The test below uses both, and the second test fails on purpose so the message of the library appears:

using FluentAssertions; using Moq; namespace MockTests; public interface IClock { string Now { get; } } public sealed class Greeting(IClock clock) { public string Text() => $"good day at {clock.Now}"; } public class GreetingTests { [Fact] public void Uses_the_clock() { var clock = new Mock<IClock>(); clock.Setup(c => c.Now).Returns("09:30"); var greeting = new Greeting(clock.Object); greeting.Text().Should().Be("good day at 09:30"); clock.Verify(c => c.Now, Times.Once); } [Fact] public void A_fluent_assertion_reports_a_difference() { var values = new[] { 1, 2, 3 }; values.Should().HaveCount(2); } }

  Determining projects to restore...
  All projects are up-to-date for restore.
  MockTests -> /tmp/r10/MockTests/bin/Debug/net10.0/MockTests.dll
Test run for /tmp/r10/MockTests/bin/Debug/net10.0/MockTests.dll (.NETCoreApp,Version=v10.0)
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:00.86]     MockTests.GreetingTests.A_fluent_assertion_reports_a_difference [FAIL]
  Failed MockTests.GreetingTests.A_fluent_assertion_reports_a_difference [237 ms]
  Error Message:
   Expected values to contain 2 item(s), but found 3: {1, 2, 3}.
  Stack Trace:
     at FluentAssertions.Collections.GenericCollectionAssertions`3.HaveCount(Int32 expected, String because, Object[] becauseArgs)
   at MockTests.GreetingTests.A_fluent_assertion_reports_a_difference() in /tmp/r10/MockTests/GreetingTests.cs:line 32
   at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

Failed!  - Failed:     1, Passed:     1, Skipped:     0, Total:     2, Duration: 582 ms - MockTests.dll (net10.0)
status: 1

The failure message is the value of FluentAssertions: Expected values to contain 2 item(s), but found 3, with the values in braces. Moq passed its verification, which is the line that is missing from the output: a mock that was not used as the test expected would have thrown MockException from Verify and failed the first test instead.

BenchmarkDotNet

A benchmark needs more than a Stopwatch: it needs a warm-up, several iterations, a statistical summary and a machine description. BenchmarkDotNet is the library that does that, and its program is a console application with a method marked [Benchmark]. The job below is a short one, one launch, one warm-up and three iterations:

using BenchmarkDotNet.Attributes; using BenchmarkDotNet.Jobs; using BenchmarkDotNet.Running; Console.WriteLine("bench 1.0"); BenchmarkRunner.Run<SumBench>(); [SimpleJob(launchCount: 1, warmupCount: 1, iterationCount: 3)] public class SumBench { [Benchmark] public long Sum() { var total = 0L; for (var i = 0; i < 1000; i++) { total += i; } return total; } }

dotnet run -c Release took 27.4 seconds, of which 20.98 seconds were the benchmark and 4.11 seconds the measured method. The summary it printed:

// * Summary *

BenchmarkDotNet v0.15.8, Linux Alpine Linux v3.24
Intel Core i3-6006U CPU 2.00GHz (Skylake), 1 CPU, 4 logical and 2 physical cores
.NET SDK 10.0.303
  [Host]     : .NET 10.0.15 (10.0.15, 10.0.1526.53008), X64 RyuJIT x86-64-v3
  Job-FGEKWY : .NET 10.0.15 (10.0.15, 10.0.1526.53008), X64 RyuJIT x86-64-v3

IterationCount=3  LaunchCount=1  WarmupCount=1  

| Method | Mean     | Error    | StdDev  |
|------- |---------:|---------:|--------:|
| Sum    | 985.2 ns | 145.9 ns | 8.00 ns |

// * Legends *
  Mean   : Arithmetic mean of all measurements
  Error  : Half of 99.9% confidence interval
  StdDev : Standard deviation of all measurements
  1 ns   : 1 Nanosecond (0.000000001 sec)

// ***** BenchmarkRunner: End *****
Run time: 00:00:04 (4.11 sec), executed benchmarks: 1

Global total time: 00:00:20 (20.98 sec), executed benchmarks: 1
// * Artifacts cleanup *
Artifacts cleanup is finished
status: 0

The summary names the runtime, the processor and the iteration counts, so a reader can judge the number. The mean of 985.2 ns is a measurement of this container on this day and moves with the load of the machine; the benchmark is also a Release build, because a Debug build of a loop measures the debugger. The run is long enough that this page shows it as a listing rather than rebuilding it with the page check, which runs every program of the other pages.

A project named after its package

The first attempt at the two templates used the folder names nunit and mstest, and NuGet stopped with an error that looks like a typo and is not one:

/tmp/r10/nunit/nunit.csproj : error NU1108: Cycle detected.
/tmp/r10/nunit/nunit.csproj : error NU1108:   nunit -> NUnit (>= 4.3.2).
  Failed to restore /tmp/r10/nunit/nunit.csproj (in 14.97 sec).

The project is nunit.csproj, and its default package id is nunit, the name of the package it references. NuGet reads that as a package that depends on itself, which is a cycle, and refuses. The fix is a project name that is not a package name: NUnitTests and MSTestTests, which are the runs shown above. The same trap waits for moq, xunit and any other folder named after a package, and it only appears on restore, never on a build that was already restored.

The packages

PackageWhat it is forVersionIn the image cache
xunitthe framework and its assertions2.9.3yes, with the template
xunit.runner.visualstudiothe adapter that lists xUnit tests for dotnet test3.1.4yes
Microsoft.NET.Test.Sdkthe test platform and the generated entry point17.14.1yes
coverlet.collectorcoverage in the cobertura format6.0.4yes
NUnitthe framework and its assertions4.3.2no, restored here
NUnit3TestAdapterthe adapter for dotnet test5.0.0no, restored here
MSTestthe Microsoft framework, as one package4.0.2no, restored here
Moqstand-ins for interfaces and classes4.21.0no, restored here
FluentAssertionsassertions written as sentences8.11.0no, restored here
BenchmarkDotNetbenchmarks with a warm-up and a summary0.15.8no, restored here

A package that is not in the cache needs the feed, and a feed that cannot be reached stops the restore with NU1301, which the xUnit page of this group shows with its real output. The status of a failed restore is one, and the tests that depend on it never run.

The group ends here. The pages before this one build a suite from a driver and a shell script to a framework and a coverage file; this page is the map of the packages around them, so that a reader can choose one and know what it costs.

Exercises

  1. Convert the xUnit test file of the previous page to NUnit and run both suites.
  2. Replace the hand-written mock of the greeting with a Moq mock and check the verification.
  3. Rename a project to the name of one of its packages on purpose and read the NU1108 error.
  4. Run BenchmarkDotNet with a CPU-bound method and with the same method wrapped in Task.Delay and compare the two summaries.

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. Which attribute does NUnit use for a test method?

    Answer

    [Test]

    The template test file of NUnit marks Test1 with [Test].

  2. Which attribute does MSTest use for a test method?

    Answer

    [TestMethod]

    The MSTest template writes [TestClass] and [TestMethod].

  3. What does this program print? Console.WriteLine("Restored in 649 ms");

    Answer

    Restored in 649 ms

    That is the restore line of the NUnit run on the page.

  4. What does Moq create?

    Answer

    A stand-in object for an interface, with the calls the test sets up

    The page mocks IClock and verifies that Now was read once.

  5. What does this program print? var values = new[] { 1, 2, 3 }; Console.WriteLine(values.Length);

    Answer

    3

    FluentAssertions reported that it found three items where two were expected.

  6. What does NU1108 report?

    Answer

    A package that depends on itself, because the project has the package name

    The measured line reads nunit -> NUnit (>= 4.3.2).

  7. Why does BenchmarkDotNet ask for a Release build?

    Answer

    A Debug build of a loop measures the debug configuration

    The page says the benchmark ran with dotnet run -c Release.

  8. What does this program print? var mean = 985.2; Console.WriteLine($"{mean:F1} ns");

    Answer

    985.2 ns

    One decimal place is the F1 format of the number.

  9. Why can the second project of a package restore without the network?

    Answer

    The package is already in the NuGet cache of the machine

    The NUnit restore took 649 ms after its packages were cached.

  10. What does a release build change about a benchmark?

    Answer

    The code is optimised, so the number describes the program and not the debugger

    Lesson 107 measured the same loop in Debug and in Release.

The code on this page was built with the .NET SDK 10.0.303 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev