Measuring performance
A performance claim is a measurement with the conditions attached. The numbers below were measured in this container, on this machine, with this compiler, and they are worth exactly that: a demonstration of how to measure, not a promise about another machine. Every number on the page came out of a command that is also on the page.
What to measure
Wall time of the whole process is the first number, because it is what the batch window and the person waiting for a report care about. It also includes the runtime start, the OPEN and CLOSE of the files and the writing of the output, which is right: a program that spends a second reading and a minute sorting is a minute, not a second.
time gives one run, and hyperfine gives many runs, a warm-up, a mean, a spread and a comparison. Both are in this image; the second is what a number worth writing down comes from. Before anything: name the machine and the compiler, because a result without them cannot be reproduced.
A benchmark and its build
The workload is deliberately small and boring: fifty million additions in a loop, with COMP-5 counters so the arithmetic is machine arithmetic and the measurement is about the loop and the compiler, not about decimal conversion.
IDENTIFICATION DIVISION.
PROGRAM-ID. BENCH.
DATA DIVISION.
WORKING-STORAGE SECTION.
01 WS-I PIC 9(9) COMP-5.
01 WS-TOTAL PIC 9(18) COMP-5.
01 WS-OUT PIC Z(17)9.
PROCEDURE DIVISION.
PERFORM VARYING WS-I FROM 1 BY 1 UNTIL WS-I IS GREATER THAN 50000000
ADD WS-I TO WS-TOTAL
END-PERFORM.
MOVE WS-TOTAL TO WS-OUT
DISPLAY FUNCTION TRIM(WS-OUT).
STOP RUN.
cobc -x -free -Wall -o bench-o0 bench.cob
./bench-o0
1250000025000000
The total is the one the arithmetic gives: fifty million times fifty million and one, over two. Every build on this page prints the same line, which is the first check of any benchmark: the fast build must compute what the slow one computed.
The optimiser
-O2 passes optimisation to the C compiler that cobc drives. The same source, two builds, fifteen runs each after a warm-up:
cobc -x -free -Wall -O2 -o bench-o2 bench.cob
./bench-o2
hyperfine --warmup 3 --runs 15 ./bench-o0 ./bench-o2
1250000025000000
Benchmark 1: ./bench-o0
Time (mean ± σ): 735.5 ms ± 201.9 ms [User: 678.9 ms, System: 3.0 ms]
Range (min … max): 609.3 ms … 1317.9 ms 15 runs
Benchmark 2: ./bench-o2
Time (mean ± σ): 31.6 ms ± 2.3 ms [User: 29.6 ms, System: 2.2 ms]
Range (min … max): 28.9 ms … 37.9 ms 15 runs
Summary
./bench-o2 ran
23.28 ± 6.62 times faster than ./bench-o0
Two numbers tell the story and a third warns about it. By the minimum, 609.3 ms against 28.9 ms is about twenty-one times. By the mean it is twenty-three, and the mean of the first build carries a spread of two hundred milliseconds, which is a third of its own value. That spread is the machine, not the program: this container shares a host, and the host was busy during some runs.
A loop of additions is the best case for an optimiser. It can see that the result does not depend on anything outside the loop and turn fifty million iterations into a constant, and the twenty times is a measure of how much arithmetic an optimiser can remove, not of what a real report will gain.
Not every program scales
The second workload fills a table of a thousand entries and then searches it for twenty thousand keys, one entry at a time, stopping at a hit. It does a million table reads and leaves the optimiser much less to remove:
IDENTIFICATION DIVISION.
PROGRAM-ID. SEARCHB.
DATA DIVISION.
WORKING-STORAGE SECTION.
01 T-AREA.
05 T-ITEM PIC 9(4) OCCURS 1000 TIMES.
01 WS-I PIC 9(4) COMP-5.
01 WS-J PIC 9(4) COMP-5.
01 WS-KEY PIC 9(4) COMP-5.
01 WS-FOUND PIC X VALUE "N".
01 WS-HITS PIC 9(9) COMP-5.
PROCEDURE DIVISION.
PERFORM VARYING WS-I FROM 1 BY 1 UNTIL WS-I IS GREATER THAN 1000
COMPUTE T-ITEM(WS-I) = WS-I * 3
END-PERFORM.
PERFORM VARYING WS-I FROM 1 BY 1 UNTIL WS-I IS GREATER THAN 20000
COMPUTE WS-KEY = FUNCTION MOD(WS-I * 7, 1000) + 1
MOVE "N" TO WS-FOUND
PERFORM VARYING WS-J FROM 1 BY 1 UNTIL WS-J IS GREATER THAN 1000
IF T-ITEM(WS-J) = WS-KEY
MOVE "Y" TO WS-FOUND
EXIT PERFORM
END-IF
END-PERFORM
IF WS-FOUND = "Y"
ADD 1 TO WS-HITS
END-IF
END-PERFORM.
DISPLAY "hits " WS-HITS.
STOP RUN.
cobc -x -free -Wall -o search-o0 search.cob
cobc -x -free -Wall -O2 -o search-o2 search.cob
./search-o0
./search-o2
hyperfine --warmup 2 --runs 8 ./search-o0 ./search-o2
hits 0000006660
hits 0000006660
Benchmark 1: ./search-o0
Time (mean ± σ): 345.4 ms ± 137.4 ms [User: 319.8 ms, System: 1.7 ms]
Range (min … max): 254.6 ms … 633.2 ms 8 runs
Benchmark 2: ./search-o2
Time (mean ± σ): 172.1 ms ± 11.6 ms [User: 170.4 ms, System: 2.0 ms]
Range (min … max): 163.8 ms … 197.4 ms 8 runs
Summary
./search-o2 ran
2.01 ± 0.81 times faster than ./search-o0
One and a half times by the minimum and two times by the mean, where the arithmetic loop got twenty. The lesson is the shape of the two results: the optimiser helps a loop it can see through, and it helps a loop with table access and a branch much less. A performance claim about a report program that quotes the arithmetic number is a claim about the wrong program.
One run is not a measurement
The same binary, three single runs with time, one after another:
/usr/bin/time -f "%e s" ./bench-o0
0.63 s
0.70 s
1.19 s
Almost a factor of two between the best and the worst run of a program that did the same work. A report of "it takes about a second" from any one of those three would be as true and as useful as a coin toss. Three habits make the number usable: run a warm-up, run many times, and quote a statistic that survives an outlier. The minimum is the most stable number on a shared machine, because noise only ever adds time; the median is the fairest when the machine is uniformly slow.
The container itself is part of the conditions. This image has two virtual cores at most, no control over the host scheduler, and no way to pin the process to a core, so a run can be interrupted by anything else on the host. A number measured here is a number about here.
Reporting a number
A sentence with a number in it should carry six facts, or it is a guess with a decimal point:
what : fifty million additions, COMP-5, no file I/O
compiler : cobc (GnuCOBOL) 3.2.0, default flags and -O2
machine : Alpine Linux 3.24.3, x86_64, Docker, shared host
method : hyperfine, 3 warm-up runs, 15 runs, wall time
statistic : min 609.3 ms and 28.9 ms, mean 735.5 ms and 31.6 ms
caveat : the arithmetic loop is the best case for -O2
With those six lines a reader can decide whether the number applies to their program, and can try to reproduce it. Without them the number is a story. The next lesson uses the same method on a change that matters more than a compiler flag: the choice between a serial search and a binary one, and between a file read per record and a single pass over the file.
Exercises
- Run the search benchmark with --runs 30. Does the ratio move closer to the minimum ratio or to the mean ratio?
- Build bench.cob with -O and with -Os and compare the two minima. Which one is faster here?
- Change the loop to one million iterations and time it. How much of the run is the process start and the runtime, rather than the loop?
- Time ./bench-o2 while another copy runs in the background. What happens to the spread, and what does that say about a number measured on a busy machine?
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
Why is the minimum a useful statistic on a shared machine?
Answer
Noise only adds time, so the fastest run is the closest to the work itself
The three single runs spread from 0.63 to 1.19 s on the same binary.
-
What did -O2 do to the arithmetic loop?
Answer
It made it about twenty times faster
609.3 ms against 28.9 ms by the minimum, and both printed the same total.
-
Why did the table search gain less from -O2?
Answer
It reads a table and branches, which an optimiser cannot remove
The search gained about 1.5 to 2 times, not twenty.
-
Which fact does a performance number NOT need?
Answer
The colour of the terminal it was run in
The other three are in the six-line report.
-
What does this program print?
IDENTIFICATION DIVISION. PROGRAM-ID. Q5. DATA DIVISION. WORKING-STORAGE SECTION. 01 I PIC 9(2). 01 N PIC 9(3) VALUE 0. PROCEDURE DIVISION. PERFORM VARYING I FROM 2 BY 2 UNTIL I IS GREATER THAN 6 ADD I TO N END-PERFORM. DISPLAY N. STOP RUN.Answer
012
Two plus four plus six is twelve.
-
What does this program print?
IDENTIFICATION DIVISION. PROGRAM-ID. Q6. DATA DIVISION. WORKING-STORAGE SECTION. 01 A PIC 9(2) VALUE 9. 01 B PIC 9(2) VALUE 4. 01 C PIC 9(2). PROCEDURE DIVISION. COMPUTE C = FUNCTION MOD(A, B) DISPLAY C. STOP RUN.Answer
01
Nine divided by four leaves one.
-
What does this program print?
IDENTIFICATION DIVISION. PROGRAM-ID. Q7. DATA DIVISION. WORKING-STORAGE SECTION. 01 W PIC X(6) VALUE "bench ". PROCEDURE DIVISION. DISPLAY "[" FUNCTION TRIM(W) "]". STOP RUN.Answer
[bench]
TRIM removes the trailing space of the six-character field.
-
What does this program print?
IDENTIFICATION DIVISION. PROGRAM-ID. Q8. DATA DIVISION. WORKING-STORAGE SECTION. 01 T. 05 T-V PIC 9(2) OCCURS 3 TIMES VALUE 7. 01 I PIC 9(2). 01 S PIC 9(3) VALUE 0. PROCEDURE DIVISION. PERFORM VARYING I FROM 1 BY 1 UNTIL I IS GREATER THAN 3 ADD T-V(I) TO S END-PERFORM. DISPLAY S. STOP RUN.Answer
021
Three entries of seven add to twenty-one.
-
Why must both builds print the same total?
Answer
A faster build that computes something else is not a measurement of the same program
Every build on the page printed 1250000025000000.
-
What is the honest limit of the numbers on this page?
Answer
They describe this compiler and this shared machine, not any other
The lesson says a number measured here is a number about here.
The code on this page was built with cobc (GnuCOBOL) 3.2.0 and timed with hyperfine on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev