Stopwatch and a benchmark

A performance claim needs a measurement, and a measurement needs a clock with a resolution finer than the work. Stopwatch reads the monotonic timer of the system, so a clock adjustment during the run does not change the result. This lesson measures one small loop and shows how to present the number honestly.

Stopwatch

StartNew starts a running watch, and Stop freezes its Elapsed. The property is a TimeSpan, and Elapsed.TotalMilliseconds is the number a table wants:

using System.Diagnostics; var watch = Stopwatch.StartNew(); var sum = 0L; for (var i = 0; i < 1000000; i++) { sum += i; } watch.Stop(); Console.WriteLine($"{watch.Elapsed.TotalMilliseconds:F2} ms {sum}"); Console.WriteLine($"ticks {watch.ElapsedTicks}"); Console.WriteLine($"running {watch.IsRunning}");

dotnet run

4.04 ms 499999500000
ticks 4043315
running False

One million additions of an int took about four milliseconds, and the tick count is the raw number the timer reports. The frequency of the timer is Stopwatch.Frequency, and ElapsedTicks divided by it is the elapsed seconds; a program that compares machines uses the milliseconds, not the ticks.

Five runs and a median

One run is noise: the JIT, the garbage collector and the other processes on the machine all move the number. The program warms the loop up once, then measures five runs of the same work and reports the minimum and the median:

using System.Diagnostics; var warm = 0L; for (var i = 0; i < 100000; i++) { warm += i; } Console.WriteLine($"warm {warm}"); var runs = new List<double>(); for (var run = 0; run < 5; run++) { var watch = Stopwatch.StartNew(); var sum = 0L; for (var i = 0; i < 1000000; i++) { sum += i; } watch.Stop(); runs.Add(watch.Elapsed.TotalMilliseconds); Console.WriteLine($"run {run + 1} {watch.Elapsed.TotalMilliseconds:F2} ms {sum}"); } var ordered = runs.Order().ToArray(); Console.WriteLine($"min {ordered[0]:F2} median {ordered[2]:F2} max {ordered[^1]:F2}");

dotnet run

warm 4999950000
run 1 4.14 ms 499999500000
run 2 5.69 ms 499999500000
run 3 4.15 ms 499999500000
run 4 4.05 ms 499999500000
run 5 4.07 ms 499999500000
min 4.05 median 4.14 max 5.69

The five runs stayed between four and six milliseconds, and the median of the ordered five is the third value, 4.14. The warm run is not part of the table: it exists to let the runtime compile the loop before the clock starts. A number without the range is a guess; a number with the minimum and the median is a measurement.

The measured numbers

The table below is one run of the second program in this container, not an average of many machines:

RunMilliseconds
14.14
25.69
34.15
44.05
54.07
min / median4.05 / 4.14

The machine was a Docker container on 4 CPUs, with .NET SDK 10.0.303 and the runtime 10.0.15, on Alpine Linux 3.24.3 x86_64. A different machine gives different numbers, and the lesson that compares two pieces of code only ever compares numbers taken on the same machine in the same session.

What a measurement says

Exercises

  1. Change the loop to two million iterations and check that the time roughly doubles.
  2. Print Stopwatch.Frequency and compute the seconds from the ticks.
  3. Run the program with

    Questions

    A new set of questions in a new order every time. Choose one answer for each, then check.

    1. What does Stopwatch measure?

      Answer

      Elapsed time from a monotonic timer

      It reads the high-resolution timer, so a clock change does not move it.

    2. Why does the benchmark run a warm-up pass first?

      Answer

      The runtime compiles and optimizes the code before the clock starts

      The first execution pays for JIT compilation, which is not the work being measured.

    3. What does this program print? using System.Diagnostics; var watch = Stopwatch.StartNew(); Console.WriteLine(watch.IsRunning);

      Answer

      True

      StartNew returns a watch that is already running.

    4. What is the median of five ordered values?

      Answer

      The third value

      Two values are below it and two above it.

    5. What does this program print? var values = new[] { 3.0, 1.0, 2.0 }; Console.WriteLine(values.Order().ElementAt(1));

      Answer

      2

      The ordered values are 1, 2 and 3, and index one is 2.

    6. Why is DateTime.Now a poor clock for a benchmark?

      Answer

      Its resolution is coarse and it can jump with the system clock

      Stopwatch is monotonic and has a much finer resolution.

    7. What does this program print? using System.Diagnostics; var watch = Stopwatch.StartNew(); watch.Stop(); Console.WriteLine(watch.IsRunning);

      Answer

      False

      Stop freezes the watch, so it is no longer running.

    8. What does the format F2 do to a double?

      Answer

      Writes it with two decimal places

      The table columns used it to keep the numbers readable.

    9. Why report a minimum and a median instead of one run?

      Answer

      A single run is noise; the range shows how stable the number is

      The five measured runs moved between 4.05 and 5.69 milliseconds.

    10. What does this program print? using System.Diagnostics; Console.WriteLine(Stopwatch.Frequency > 0);

      Answer

      True

      The frequency is the ticks per second of the timer, and it is positive.

    The code on this page was built with the .NET SDK 10.0.303 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.

    Article author: Arthur Isaev