Stopwatch and a benchmark
A performance claim needs a measurement, and a measurement needs a clock with a resolution finer than the work. Stopwatch reads the monotonic timer of the system, so a clock adjustment during the run does not change the result. This lesson measures one small loop and shows how to present the number honestly.
Stopwatch
StartNew starts a running watch, and Stop freezes its Elapsed. The property is a TimeSpan, and Elapsed.TotalMilliseconds is the number a table wants:
using System.Diagnostics;
var watch = Stopwatch.StartNew();
var sum = 0L;
for (var i = 0; i < 1000000; i++)
{
sum += i;
}
watch.Stop();
Console.WriteLine($"{watch.Elapsed.TotalMilliseconds:F2} ms {sum}");
Console.WriteLine($"ticks {watch.ElapsedTicks}");
Console.WriteLine($"running {watch.IsRunning}");
dotnet run
4.04 ms 499999500000
ticks 4043315
running False
One million additions of an int took about four milliseconds, and the tick count is the raw number the timer reports. The frequency of the timer is Stopwatch.Frequency, and ElapsedTicks divided by it is the elapsed seconds; a program that compares machines uses the milliseconds, not the ticks.
Five runs and a median
One run is noise: the JIT, the garbage collector and the other processes on the machine all move the number. The program warms the loop up once, then measures five runs of the same work and reports the minimum and the median:
using System.Diagnostics;
var warm = 0L;
for (var i = 0; i < 100000; i++)
{
warm += i;
}
Console.WriteLine($"warm {warm}");
var runs = new List<double>();
for (var run = 0; run < 5; run++)
{
var watch = Stopwatch.StartNew();
var sum = 0L;
for (var i = 0; i < 1000000; i++)
{
sum += i;
}
watch.Stop();
runs.Add(watch.Elapsed.TotalMilliseconds);
Console.WriteLine($"run {run + 1} {watch.Elapsed.TotalMilliseconds:F2} ms {sum}");
}
var ordered = runs.Order().ToArray();
Console.WriteLine($"min {ordered[0]:F2} median {ordered[2]:F2} max {ordered[^1]:F2}");
dotnet run
warm 4999950000
run 1 4.14 ms 499999500000
run 2 5.69 ms 499999500000
run 3 4.15 ms 499999500000
run 4 4.05 ms 499999500000
run 5 4.07 ms 499999500000
min 4.05 median 4.14 max 5.69
The five runs stayed between four and six milliseconds, and the median of the ordered five is the third value, 4.14. The warm run is not part of the table: it exists to let the runtime compile the loop before the clock starts. A number without the range is a guess; a number with the minimum and the median is a measurement.
The measured numbers
The table below is one run of the second program in this container, not an average of many machines:
| Run | Milliseconds |
|---|---|
| 1 | 4.14 |
| 2 | 5.69 |
| 3 | 4.15 |
| 4 | 4.05 |
| 5 | 4.07 |
| min / median | 4.05 / 4.14 |
The machine was a Docker container on 4 CPUs, with .NET SDK 10.0.303 and the runtime 10.0.15, on Alpine Linux 3.24.3 x86_64. A different machine gives different numbers, and the lesson that compares two pieces of code only ever compares numbers taken on the same machine in the same session.
What a measurement says
- It says what this program did on this machine with this build, and nothing more.
- A Debug build is slower than a Release build, which the lesson on the two configurations measures; a benchmark states the configuration.
- A short run is dominated by startup and by the JIT; the tiered compilation lesson shows the first runs of a hot loop getting faster.
- A container shares the CPU with other processes, so the same program run twice gives two different tables; that is why the table shows every run.
Exercises
- Change the loop to two million iterations and check that the time roughly doubles.
- Print Stopwatch.Frequency and compute the seconds from the ticks.
-
Run the program with
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
What does Stopwatch measure?
Answer
Elapsed time from a monotonic timer
It reads the high-resolution timer, so a clock change does not move it.
-
Why does the benchmark run a warm-up pass first?
Answer
The runtime compiles and optimizes the code before the clock starts
The first execution pays for JIT compilation, which is not the work being measured.
-
What does this program print?
using System.Diagnostics; var watch = Stopwatch.StartNew(); Console.WriteLine(watch.IsRunning);Answer
True
StartNew returns a watch that is already running.
-
What is the median of five ordered values?
Answer
The third value
Two values are below it and two above it.
-
What does this program print?
var values = new[] { 3.0, 1.0, 2.0 }; Console.WriteLine(values.Order().ElementAt(1));Answer
2
The ordered values are 1, 2 and 3, and index one is 2.
-
Why is DateTime.Now a poor clock for a benchmark?
Answer
Its resolution is coarse and it can jump with the system clock
Stopwatch is monotonic and has a much finer resolution.
-
What does this program print?
using System.Diagnostics; var watch = Stopwatch.StartNew(); watch.Stop(); Console.WriteLine(watch.IsRunning);Answer
False
Stop freezes the watch, so it is no longer running.
-
What does the format F2 do to a double?
Answer
Writes it with two decimal places
The table columns used it to keep the numbers readable.
-
Why report a minimum and a median instead of one run?
Answer
A single run is noise; the range shows how stable the number is
The five measured runs moved between 4.05 and 5.69 milliseconds.
-
What does this program print?
using System.Diagnostics; Console.WriteLine(Stopwatch.Frequency > 0);Answer
True
The frequency is the ticks per second of the timer, and it is positive.
The code on this page was built with the .NET SDK 10.0.303 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev
-