Measuring performance
A timing is a measurement, and a measurement is only worth something when the conditions are stated: which clock, how many runs, what was warmed up, and what the loop actually did. The standard library has the clock and the arithmetic; the method is yours.
This lesson measures one small loop with steady_clock, converts the duration to microseconds, repeats the run and keeps the best. The numbers are from this machine and change with the load, so the page shows what the program prints, not a claim about speed.
A measurement taken on a loaded machine is a measurement of the load. A benchmark should record how many runs it made, which build it timed and on which machine, or the number cannot be reproduced and cannot be compared with the next one.
The chrono types are also what a log line should carry: a duration in a named unit reads better than a raw count of ticks, and the type keeps the unit attached to the number.
steady_clock around a loop
std::chrono::steady_clock is monotonic: it never goes backwards and is not adjusted by the system time. now() gives a time point, and the difference of two time points is a duration.
The program sums i % 7 ten million times, measures the loop and prints the sum and whether the elapsed time was positive. The sum is consumed by the print, which is what stops the optimiser from removing the loop.
duration_cast converts the duration to microseconds, and count() gives the number. The cast truncates, so a loop shorter than a microsecond reports 0, which is the first sign that the measurement is too small to mean anything.
#include <chrono>
#include <iostream>
int main()
{
auto start = std::chrono::steady_clock::now();
long total = 0;
for (int i = 0; i < 10000000; ++i) {
total += i % 7;
}
auto end = std::chrono::steady_clock::now();
auto micros = std::chrono::duration_cast<std::chrono::microseconds>(end - start).count();
std::cout << total << ' ' << (micros > 0) << '\n';
}
g++ -std=c++23 -Wall -Wextra clock.cpp -o clock_measure
./clock_measure
29999994 1
Warm up and repeat
The first run of a loop pays for cold caches, an unpopulated branch predictor and a page fault or two. A measurement that matters runs the loop once to warm up and then times several runs, keeping the best or the median.
The program times the same loop three times and prints the best and the worst in microseconds. The best is the least disturbed run, and the gap between the two is how much the machine's noise costs.
The minimum is the usual statistic for a micro-benchmark because it is the run with the fewest interruptions. The average is the number to report when the workload is a service and the interruptions are part of it.
#include <algorithm>
#include <chrono>
#include <iostream>
#include <vector>
long run()
{
long total = 0;
for (int i = 0; i < 1000000; ++i) {
total += i % 7;
}
return total;
}
int main()
{
volatile long sink = run();
(void)sink;
std::vector<long> times;
for (int attempt = 0; attempt < 3; ++attempt) {
auto start = std::chrono::steady_clock::now();
volatile long result = run();
auto end = std::chrono::steady_clock::now();
(void)result;
times.push_back(
std::chrono::duration_cast<std::chrono::microseconds>(end - start).count());
}
std::sort(times.begin(), times.end());
std::cout << times.front() << ' ' << times.back() << ' ';
std::cout << (times.front() >= 0) << '\n';
}
g++ -std=c++23 -Wall -Wextra repeat.cpp -o repeat
./repeat
4988 5681 1
The clock that is not steady
system_clock is the wall clock: it can jump when the machine synchronises its time, so the difference of two readings can even be negative. steady_clock exists for durations, and high_resolution_clock is an alias of one of them.
The program prints the is_steady flag of the three clocks. On this build it is 1 for steady_clock and 0 for both system_clock and high_resolution_clock, which is an alias of system_clock in libstdc++.
A timestamp for a log line is a system_clock job; an interval is a steady_clock job. Mixing the two is the mistake of the next section, and the type system catches it at compile time.
#include <chrono>
#include <iostream>
int main()
{
std::cout << std::chrono::steady_clock::is_steady << ' ';
std::cout << std::chrono::system_clock::is_steady << ' ';
std::cout << std::chrono::high_resolution_clock::is_steady << '\n';
}
g++ -std=c++23 -Wall -Wextra steady.cpp -o steady_clocks
./steady_clocks
1 0 0
Two clocks do not subtract
A duration comes from subtracting two time points of the same clock. Subtracting a steady_clock point from a system_clock point has no meaning, and the library makes it a compile error instead of a number.
The program takes the start from one clock and the end from the other. The compiler reports that no subtraction matches the two time_point types, and the message names both clocks in the template arguments.
The fix is to take both readings from the same clock. When the two really are needed, each interval is computed inside its own clock and the results are compared as durations, never as time points.
#include <chrono>
int main()
{
auto start = std::chrono::steady_clock::now();
auto end = std::chrono::system_clock::now();
auto elapsed = std::chrono::duration_cast<std::chrono::milliseconds>(end - start);
return static_cast<int>(elapsed.count());
}
g++ -std=c++23 -Wall -Wextra mixed.cpp -o mixed
mixed.cpp: In function 'int main()':
mixed.cpp:8:78: error: no match for 'operator-' (operand types are 'std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1, 1000000000> > >' and 'std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long int, std::ratio<1, 1000000000> > >')
8 | auto elapsed = std::chrono::duration_cast<std::chrono::milliseconds>(end - start);
| ~~~ ^ ~~~~~
| | |
| | time_point<std::chrono::_V2::steady_clock,[...]>
| time_point<std::chrono::_V2::system_clock,[...]>
mixed.cpp:8:78: note: there are 5 candidates
8 | auto elapsed = std::chrono::duration_cast<std::chrono::milliseconds>(end - start);
| ~~~~^~~~~~~
In file included from /usr/include/c++/15.2.0/string:50,
from /usr/include/c++/15.2.0/bits/locale_classes.h:42,
from /usr/include/c++/15.2.0/bits/ios_base.h:43,
from /usr/include/c++/15.2.0/ios:46,
from /usr/include/c++/15.2.0/istream:42,
from /usr/include/c++/15.2.0/sstream:42,
from /usr/include/c++/15.2.0/chrono:51,
from mixed.cpp:1:
/usr/include/c++/15.2.0/bits/stl_iterator.h:620:5: note: candidate 1: 'template<class _IteratorL, class _IteratorR> constexpr decltype ((__y.base() - __x.base())) std::operator-(const reverse_iterator<_IteratorL>&, const reverse_iterator<_IteratorR>&)'
620 | operator-(const reverse_iterator<_IteratorL>& __x,
| ^~~~~~~~
/usr/include/c++/15.2.0/bits/stl_iterator.h:620:5: note: template argument deduction/substitution failed:
mixed.cpp:8:80: note: 'std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1, 1000000000> > >' is not derived from 'const std::reverse_iterator<_IteratorL>'
8 | auto elapsed = std::chrono::duration_cast<std::chrono::milliseconds>(end - start);
| ^~~~~
/usr/include/c++/15.2.0/bits/stl_iterator.h:1785:5: note: candidate 2: 'template<class _IteratorL, class _IteratorR> constexpr decltype ((__x.base() - __y.base())) std::operator-(const move_iterator<_IteratorL>&, const move_iterator<_IteratorR>&)'
1785 | operator-(const move_iterator<_IteratorL>& __x,
| ^~~~~~~~
/usr/include/c++/15.2.0/bits/stl_iterator.h:1785:5: note: template argument deduction/substitution failed:
mixed.cpp:8:80: note: 'std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1, 1000000000> > >' is not derived from 'const std::move_iterator<_IteratorL>'
8 | auto elapsed = std::chrono::duration_cast<std::chrono::milliseconds>(end - start);
| ^~~~~
In file included from /usr/include/c++/15.2.0/chrono:45:
/usr/include/c++/15.2.0/bits/chrono.h:714:7: note: candidate 3: 'template<class _Rep1, class _Period1, class _Rep2, class _Period2> constexpr typename std::common_type<std::chrono::duration<_Rep1, _Period1>, std::chrono::duration<_Rep2, _Period2> >::type std::chrono::operator-(const duration<_Rep1, _Period1>&, const duration<_Rep2, _Period2>&)'
714 | operator-(const duration<_Rep1, _Period1>& __lhs,
| ^~~~~~~~
/usr/include/c++/15.2.0/bits/chrono.h:714:7: note: template argument deduction/substitution failed:
mixed.cpp:8:80: note: 'std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long int, std::ratio<1, 1000000000> > >' is not derived from 'const std::chrono::duration<_Rep1, _Period1>'
8 | auto elapsed = std::chrono::duration_cast<std::chrono::milliseconds>(end - start);
| ^~~~~
/usr/include/c++/15.2.0/bits/chrono.h:1133:7: note: candidate 4: 'template<class _Clock, class _Dur1, class _Rep2, class _Period2> constexpr std::chrono::time_point<_Clock, typename std::common_type<_Dur1, std::chrono::duration<_Rep2, _Period2> >::type> std::chrono::operator-(const time_point<_Clock, _Duration1>&, const duration<_Rep2, _Period2>&)'
1133 | operator-(const time_point<_Clock, _Dur1>& __lhs,
| ^~~~~~~~
/usr/include/c++/15.2.0/bits/chrono.h:1133:7: note: template argument deduction/substitution failed:
mixed.cpp:8:80: note: 'std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long int, std::ratio<1, 1000000000> > >' is not derived from 'const std::chrono::duration<_Rep2, _Period2>'
8 | auto elapsed = std::chrono::duration_cast<std::chrono::milliseconds>(end - start);
| ^~~~~
/usr/include/c++/15.2.0/bits/chrono.h:1145:7: note: candidate 5: 'template<class _Clock, class _Dur1, class _Dur2> constexpr typename std::common_type<_Duration1, _Duration2>::type std::chrono::operator-(const time_point<_Clock, _Duration1>&, const time_point<_Clock, _Duration2>&)'
1145 | operator-(const time_point<_Clock, _Dur1>& __lhs,
| ^~~~~~~~
/usr/include/c++/15.2.0/bits/chrono.h:1145:7: note: template argument deduction/substitution failed:
mixed.cpp:8:80: note: deduced conflicting types for parameter '_Clock' ('std::chrono::_V2::system_clock' and 'std::chrono::_V2::steady_clock')
8 | auto elapsed = std::chrono::duration_cast<std::chrono::milliseconds>(end - start);
| ^~~~~
Exercises
- Build the timing program with -O2 and with -O0 and compare the two measurements of the same loop.
- Add a fourth run to the repeat program and print the median instead of the best.
- Measure a loop that does nothing and explain what the number means.
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
Why is steady_clock the right clock for measuring a duration?
Answer
It is monotonic and never adjusted by the system time
A duration needs a clock that cannot jump, and system_clock can.
-
What does
duration_castdo?Answer
It converts a duration to another unit, truncating
The cast changes the unit of the duration and drops the fraction below it.
-
Why repeat a measurement and keep the best run?
Answer
The best run is the one least disturbed by the machine
Noise only adds time, so the minimum is the closest to the cost of the loop itself.
-
What is a warm-up run for?
Answer
To fill the caches and the branch predictor before timing
The first run pays costs that later runs do not, and it would distort the measurement.
-
What does this program print?
#include <iostream> int main() { long total = 0; for (int i = 0; i < 10000000; ++i) total += i % 7; std::cout << total; }Answer
29999994
The loop adds i % 7 for ten million values, and the total is 29999994.
-
Which clock does the standard define as monotonic?
Answer
std::chrono::steady_clockOnly steady_clock is specified never to go backwards; the wall clock can be stepped in either direction.
-
What does a measurement of 0 microseconds usually mean?
Answer
That the measured work is shorter than the resolution of the clock
duration_cast truncates, so a sub-microsecond loop reports zero and needs more work or a finer unit.
-
What happens with
end - startwhen the two time points come from different clocks?Answer
It does not compile: no subtraction matches the two types
The subtraction is defined within one clock, so the mixed types have no operator.
-
What does
count()on a duration return?Answer
The number of ticks in the duration's unit
A duration is a count of ticks of its period, and count gives that number.
-
Why is the result of the loop printed or stored in a volatile?
Answer
So the optimiser cannot remove the work that is being measured
Work with no visible effect may be deleted, and then the measurement times nothing.
The code on this page was built with g++ 15.2.0 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container. The programs whose section shows a compiler error were compiled until that error and were not run.
Article author: Arthur Isaev