The same kernel, timed
The pages before this one compared two programs doing one thing. This one is the benchmark a team runs before choosing: the same kernel, written the way each language wants it, at the two optimisation levels that matter, with five runs each and the minimum and median reported. The kernel is a 300 by 300 matrix multiply, 27 million multiply-adds, small enough to repeat in a second and large enough to be real work.
The kernel
Both programs fill a 300 by 300 matrix from its indices so that the two matrices are the same numbers, multiply, and print one element of the product and the time of the multiply. The loops are written in the order each language wants: j-k-i in Fortran, where i is the contiguous index, and i-k-j in C++, where j is. The two orders are the same algorithm seen through the two memory layouts, which is what makes the comparison fair.
The Fortran program
The file mm.f90. The argument intrinsic switches the three loops for MATMUL, which is the comparison in the section after the table:
program mm_f
use, intrinsic :: iso_fortran_env, only: dp => real64
implicit none
integer, parameter :: n = 300
real(dp), allocatable :: a(:, :), b(:, :), c(:, :)
integer :: i, j, k, t0, t1, rate
character(len=16) :: arg
call get_command_argument(1, arg)
allocate(a(n, n), b(n, n), c(n, n))
do j = 1, n
do i = 1, n
a(i, j) = real(mod(i + j, 7) + 1, dp)
b(i, j) = real(mod(i * j, 5) + 1, dp)
end do
end do
call system_clock(t0, rate)
if (trim(arg) == 'intrinsic') then
c = matmul(a, b)
else
c = 0.0_dp
do j = 1, n
do k = 1, n
do i = 1, n
c(i, j) = c(i, j) + a(i, k) * b(k, j)
end do
end do
end do
end if
call system_clock(t1)
print '(a,es18.10,a,es10.3,a)', 'fortran c(1,1)=', c(1, 1), ' time=', real(t1 - t0) / real(rate), ' s'
end program mm_f
gfortran -Wall -Wextra -std=f2018 -O2 mm.f90 -o mm_f2
./mm_f2
The C++ program
The file mm.cpp:
#include <chrono>
#include <cstdio>
#include <cstdlib>
#include <vector>
int main() {
const int n = 300;
std::vector<double> a(static_cast<size_t>(n) * n);
std::vector<double> b(static_cast<size_t>(n) * n);
std::vector<double> c(static_cast<size_t>(n) * n, 0.0);
for (int i = 1; i <= n; ++i) {
for (int j = 1; j <= n; ++j) {
a[static_cast<size_t>(i - 1) * n + (j - 1)] = ((i + j) % 7) + 1;
b[static_cast<size_t>(i - 1) * n + (j - 1)] = ((i * j) % 5) + 1;
}
}
const auto start = std::chrono::steady_clock::now();
for (int i = 0; i < n; ++i) {
for (int k = 0; k < n; ++k) {
const double aik = a[static_cast<size_t>(i) * n + k];
for (int j = 0; j < n; ++j) {
c[static_cast<size_t>(i) * n + j] += aik * b[static_cast<size_t>(k) * n + j];
}
}
}
const auto stop = std::chrono::steady_clock::now();
const double secs = std::chrono::duration<double>(stop - start).count();
std::printf("c++ c(1,1)=%18.10e time=%10.3e s\n", c[0], secs);
return 0;
}
g++ -std=c++23 -O2 -Wall -Wextra mm.cpp -o mm_c2
./mm_c2
Six measurements
| program | flags | min | median |
|---|---|---|---|
| Fortran, three loops | -O2 | 23.0 ms | 23.0 ms |
| Fortran, three loops | -O3 | 12.0 ms | 12.0 ms |
| C++, three loops | -O2 | 21.7 ms | 21.8 ms |
| C++, three loops | -O3 | 10.7 ms | 10.7 ms |
| Fortran, MATMUL | -O2 | 5.0 ms | 5.0 ms |
| Fortran, MATMUL | -O3 | 4.0 ms | 5.0 ms |
GNU Fortran (Alpine 15.2.0) 15.2.0 and G++ (Alpine 15.2.0) 15.2.0 on an Intel Core i3-6006U at 2.00 GHz with 4 logical CPUs (2 cores) and a 3 MB cache, inside a Docker container that shares a host. Five runs per row, min and median; the Fortran timings come from system_clock, whose resolution here is one millisecond, which is why the last digit is a one or a zero.
The three loops are within ten per cent of each other in the two languages at both levels, with C++ slightly ahead: 21.7 against 23.0 at -O2 and 10.7 against 12.0 at -O3. Two compilers from the same GCC release, the same algorithm, the same machine: the difference is the front end and the loop the optimiser sees, and on this kernel it is worth about ten per cent, not a factor.
The intrinsic
MATMUL at -O2 takes 5.0 ms where the Fortran loops take 23.0, and at -O3 4.0 to 5.0 where the loops take 12.0. It is not a better loop: gfortran calls a library routine that blocks the multiplication for the cache, and the gain is the gain of blocking, which grows with the matrix.
The C++ program has no equivalent to time. The standard library has no matrix multiply, and this image has no BLAS: a C++ project that wants the same speed links a library, which is one more dependency, one more build step and one more version to keep. The Fortran program gets it from the language, and that is the strongest single argument for Fortran in a numerical code base: the good version of the inner loop is spelled c = matmul(a, b) and cannot drift.
What the numbers do not show
Four limits are worth stating before anyone quotes this table. The matrix is 300 by 300, which is small: at 2000 the blocked routine in MATMUL would win by more, because it is the blocking that pays at that size, and the three loops in either language would need their own blocking to compete. The compiler is the same GCC back end, so this is not a comparison of GCC against clang or Intel. The machine is a two-core laptop CPU inside a shared container, which makes every number a lower bound on a real server. And a matrix multiply is one kernel: a stencil, a sparse product or a reduction has a different balance of memory and arithmetic and could rank the two languages differently.
What the table does show is the honest summary of this group: for the same algorithm written with the same care, the language is worth about ten per cent, and the library that comes with the language is worth a factor of five. A team that picks Fortran for the intrinsic and the array syntax is making a defensible choice; a team that picks it because "Fortran is faster" has not measured the same loops.
Exercises
- Change n to 1000 in both programs and run all six measurements. Which row gains the most from the size, and why?
- Write a blocked version of the three loops in C++ with 32 by 32 blocks and time it against MATMUL. How close does it get?
- Compile both with -O2 -march=native. Does the gap between the languages change?
- Time the fill as well as the multiply in both programs. Which fraction of the run time is the fill, and what does that say about micro-benchmarks?
The programs on this page were built with GNU Fortran (Alpine 15.2.0) 15.2.0 and G++ (Alpine 15.2.0) 15.2.0 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container of 4 logical CPUs, on an Intel Core i3-6006U at 2.00 GHz with a 3 MB cache. Every timing is five runs of the same binary, with min and median stated.
Article author: Arthur Isaev