Measuring performance
The machine line
A timing without the machine and the flags is a rumour. Every number on this page comes from this line:
- opencode-agent-native on Alpine Linux 3.24.3, x86_64, 4 CPUs (nproc prints 4).
- GHC 9.10.3, cabal-install 3.16.0.0.
- ghc -Wall -O2 -rtsopts for the program, and the default GC settings: no -A, no -H, no -N.
The container may be shared with other work, so the numbers are the numbers of a busy machine and not a laboratory result. That is exactly why the method below matters more than the values: five runs, a warm-up, the minimum and the median, and a statement of the flags.
The program and its flags
module Main (main) where
main :: IO ()
main = print (sum [1 .. 500000000 :: Int])
ghc -Wall -O2 -rtsopts -o bench bench.hs
[1 of 2] Compiling Main ( bench.hs, bench.o )
[2 of 2] Linking bench
The loop is the smallest benchmark that measures something: 500 million additions and one print. -O2 is the level a benchmark should use, because it is the best the compiler can do and a regression at -O2 is a regression that matters. -rtsopts is needed for the RTS flags of the last section; it does not change the code.
/usr/bin/time and five runs
The first run is a warm-up and is not counted: the binary is read from disk and the page cache is cold. Then five runs, each with /usr/bin/time -f "%e %M", which prints the wall time in seconds and the maximum resident set in kilobytes:
./bench >/dev/null
for i in 1 2 3 4 5; do /usr/bin/time -f "%e %M" ./bench >/dev/null; done
0.28 1792
0.26 1792
0.32 1792
0.27 1792
0.27 1792
The output of the program is discarded with >/dev/null, so the two columns are the time and the memory and nothing else. The resident set is the same in every run, because the program allocates a bounded amount and the RTS does not grow; the time moves by 20 percent, which is what a shared machine does.
The minimum and the median
Five values are enough for two numbers: the minimum, which is the closest to the cost of the work with no interference, and the median, which is the typical run. The mean is not reported, because one slow run moves it and the distribution here is not symmetric:
sorted: 0.26 0.27 0.27 0.28 0.32
minimum: 0.26 s
median: 0.27 s
maxrss: 1792 KB = 1.75 MiB
A report of this benchmark should quote the minimum and the median with the machine line and the flags, and say that the container was shared. A single run of 0.32 would suggest the program got slower, and a single run of 0.26 that it got faster; five runs and two statistics are the smallest honest answer.
+RTS -s from the same build
The same binary with +RTS -s reports the heap instead of the clock, and the two numbers together are what a profile of a loop needs:
./bench +RTS -s
125000000250000000
50,680 bytes allocated in the heap
3,272 bytes copied during GC
44,328 bytes maximum residency (1 sample(s))
25,304 bytes maximum slop
6 MiB total memory in use (0 MiB lost due to fragmentation)
Fifty kilobytes allocated for 500 million additions: the optimiser turned the list into a loop and the loop into two registers, so there is no list and no thunk. That is also why the elapsed time is a quarter of a second: the work is about two additions per nanosecond, which is the speed of a tight loop on one core. A version that allocated per element would show tens of megabytes here and be a different question.
What a number is worth
- Compare like with like: same flags, same input, same machine, same run length, and the same GHC version.
- Five runs, minimum and median, and the machine line next to them. If the difference between two versions is smaller than the spread of the five runs, it is not a difference.
- Use +RTS -s to tell a slow loop from a leaking one: time is the clock and residency is the heap, and they have different fixes.
- Do not benchmark the compiler: -O2 for a benchmark, -O0 for a debugger, and say which one you used.
The next lesson starts Part 7 with the advanced types: GADTs, type families and rank-N types, where the type system checks more of the program than the classes of Part 4 could.
Exercises
- Run the five-run loop on your machine and compare the median with the one here.
- Build the same program with -O1 and compare both statistics.
- Replace the range with a list built by map and reverse, then read the residency of each version.
- Measure a program of your own, with a warm-up and five runs.
- Explain why the mean is a poor statistic for five runs on a shared machine.
- Find the -A flag in the GHC user guide and say what it changes.
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
Why is the first run not counted?
Answer
It warms the page cache and the machine, so it is not typical
A cold start measures the disk and the loader as well as the program.
-
Which pair of statistics does the lesson report?
Answer
The minimum and the median
The minimum is the least disturbed run and the median is the typical one.
-
What does
/usr/bin/time -f "%e %M"print?Answer
Wall time in seconds and maximum resident set in KB
%e is the elapsed time and %M the maximum resident set size.
-
What does this program print?
module Main (main) where main :: IO () main = print (sum [1 .. 4 :: Int])Answer
10
The sum of 1 to 4 is 10.
-
What does a tiny allocation count with a quarter-second run tell you?
Answer
The loop is tight and the optimiser removed the list
Fifty kilobytes for 500 million additions means no per-element allocation, so the list is gone.
-
What does this program print?
module Main (main) where main :: IO () main = print (product [1, 2, 3 :: Int])Answer
6
The product of 1, 2 and 3 is 6.
-
Why do two versions need the same flags to be compared?
Answer
A different optimisation level is a different program
The same source at -O0 and -O2 can differ by an order of magnitude, so the level is part of the measurement.
-
What is the honest way to report a five-run benchmark?
Answer
The minimum and the median, with the machine line and the flags
The two statistics and the context are what make the number reproducible.
-
What does this program print?
module Main (main) where main :: IO () main = print (length [1 .. 100000 :: Int])Answer
100000
The range 1 to 100000 has 100000 elements.
-
What is a difference smaller than the spread of the runs?
Answer
Not evidence of a change
Noise of that size is what the same binary produces on repeated runs.
The code on this page was built with GHC 9.10.3 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev