OpenMP: parallel do

OpenMP is the other way Fortran goes parallel, and it is the one this image can actually run: the directives are comments to a compiler that does not know them, the library is in the gcc package, and a loop over four threads works out of the box. This lesson takes a loop that evaluates a function sixteen million times, builds it twice, and reports what the wall clock said over five runs each on this machine.

The directive

!$omp parallel do is a comment that starts with !$omp. A compiler without OpenMP sees a comment and compiles the loop serially; with -fopenmp the same line is a directive that turns the loop into a parallel one. That is why the same source is both the serial and the parallel program, and why a page can show both builds from one listing.

The file omp.f90:

program ompfill use, intrinsic :: iso_fortran_env, only: real64 use omp_lib implicit none integer, parameter :: n = 16000000 real(real64), allocatable :: a(:) real(real64) :: h, x, total integer :: i, t0, t1, rate, threads allocate(a(n)) h = 1.0_real64 / real(n, real64) threads = 1 !$ threads = omp_get_max_threads() call system_clock(t0, rate) !$omp parallel do private(x) do i = 1, n x = (real(i, real64) - 0.5_real64) * h a(i) = exp(-x * x) * cos(3.0_real64 * x) end do !$omp end parallel do total = sum(a) call system_clock(t1, rate) print '(a,i0)', 'threads = ', threads print '(a,f10.7)', 'integral = ', total * h print '(a,f10.1,a)', 'elapsed = ', real(t1 - t0) / real(rate) * 1000.0, ' ms' end program ompfill

The program

The loop fills an array with values of a smooth function, then sum(a) reduces it serially. The parallel part is a pure fill: element i depends only on i, so the iterations can run in any order on any thread. private(x) gives each thread its own x, because a variable declared in the procedure would otherwise be shared and every thread would write over the others.

system_clock measures wall-clock time in ticks, and the second argument is the number of ticks per second. The timing covers the fill and the serial sum, because the sum is part of the work: a program that timed only the parallel loop would report a speedup its user never sees.

Two builds

The parallel build is the one with the flag, and four threads is this machine's default:

gfortran -Wall -Wextra -std=f2018 -O2 -fopenmp omp.f90 -o omp
OMP_NUM_THREADS=4 ./omp

threads = 4
integral = 0.1725083
elapsed = 263.0 ms

The same file without the flag is a serial program, and the directive lines are comments:

gfortran -Wall -Wextra -std=f2018 -O2 omp.f90 -o omp_serial
./omp_serial

threads = 1
integral = 0.1725083
elapsed = 622.0 ms

The two integrals are equal to the last printed digit, which is the first thing to check: a parallel version that computes a different number is a parallel version with a race, and the timing of a wrong answer is not a result. The elapsed times are one run each, and a single run is not a measurement; the table below is.

The measured table

Machine: Intel(R) Core(TM) i3-6006U CPU @ 2.00GHz, 4 logical CPUs from nproc, which is two physical cores with hyper-threading (cpu cores: 2, siblings: 4 in /proc/cpuinfo), 4 GB of RAM in the container, Alpine Linux 3.24.3, GNU Fortran (Alpine 15.2.0) 15.2.0, flags -O2 and -fopenmp as shown above. Five runs of each configuration, in milliseconds:

threadsfive runsminmedianspeedup on min
1, serial binary611, 600, 567, 878, 5445446001.00
1, OpenMP build562, 562, 582, 567, 5685625670.97
2347, 306, 434, 302, 4083023471.80
4330, 280, 264, 268, 3452642802.06

Three things in the table are worth reading carefully. The OpenMP build with one thread is as fast as the serial build, within the noise: the directives cost nothing when there is nothing to share. Two threads give most of the gain, 1.8 times, and four threads add only a little on top, because there are two physical cores and the other two logical CPUs share them. And the spread between the runs is large, up to 60 per cent, because the container shares a host: the minimum is the number to compare and the median is the number to quote.

A second lesson is hidden in the same table. A program that evaluates exp and cos sixteen million times spends most of its time in the maths library, which is not parallelised; the loop that only multiplies would show a different ratio. A speedup is a property of a program on a machine, never of a compiler flag.

OMP_NUM_THREADS

The thread count is a run-time setting, not a compile-time one: OMP_NUM_THREADS=2 ./omp runs the same binary on two threads. That is why a program prints what it actually got rather than what it asked for, and omp_get_max_threads() is the enquiry. Inside a parallel region the number of the calling thread is omp_get_thread_num(), which is what a program uses to give each thread a slice of an array it fills by hand.

A library that starts its own threads and a program that sets a thread count are the same setting, and a numerical library built with OpenMP will use as many threads as the environment says. That is a surprise in a batch job that runs one process per core and finds each of them starting four threads.

omp_lib without the flag

The module omp_lib is found without the flag, because gcc ships it with the compiler, and the library is not linked:

program badomp use omp_lib implicit none print *, omp_get_max_threads() end program badomp

gfortran -Wall -Wextra -std=f2018 badomp.f90 -o badomp

/usr/lib/gcc/x86_64-alpine-linux-musl/15.2.0/../../../../x86_64-alpine-linux-musl/bin/ld: /tmp/ccbLdfPp.o: in function `MAIN__':
badomp.f90:(.text+0x56): undefined reference to `omp_get_max_threads_'
collect2: error: ld returned 1 exit status

The name in the error has an underscore at the end because that is the symbol the Fortran compiler emits for omp_get_max_threads. Without -fopenmp the linker is never told about libgomp, so the call has no code behind it. The fix is the flag, which both links the library and enables the directives.

What a parallel do requires

The iterations must be independent: no iteration may read what another writes, and no variable that changes must be shared. A loop that carries a value from one iteration to the next, a cumulative sum, a recurrence, cannot be parallelised by a directive, and the compiler will not tell you: it produces a fast wrong answer. The reduction clause of the next lesson covers one common case, and the rest needs a different algorithm.

Two more rules are worth writing down. A variable declared inside the loop body is private to the iteration and needs no clause; one declared outside is shared unless it is named in private or firstprivate. And the result must not depend on the order of the iterations: a sum of floating-point values in a different order is a different number in the last bits, which is why a parallel sum needs the reduction clause rather than a shared accumulator.

Exercises

  1. Change the loop body to a(i) = exp(-x * x) and rerun the table. Is the speedup larger or smaller, and why?
  2. Remove private(x) and run the program with two threads. What happens to the integral?
  3. Time the serial sum(a) separately with a second pair of system_clock calls. How much of the elapsed time is it?
  4. Run the same binary with OMP_NUM_THREADS=3. What does the program print, and what does the elapsed time say about the two physical cores?

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. What is a !$omp parallel do line to a compiler without -fopenmp?

    Answer

    A comment

    That is why one source is both programs.

  2. Why does x need private(x) in the loop?

    Answer

    It is declared outside the loop, so it would be shared by the threads

    Each thread must write its own temporary.

  3. What did four threads give against one thread on this machine?

    Answer

    About 2.1 times, because there are two physical cores

    The table shows min 264 against 562.

  4. What does OMP_NUM_THREADS set?

    Answer

    The number of threads the program starts with, at run time

    The same binary can run on one or four threads.

  5. What does a single timing run prove?

    Answer

    Almost nothing; a table of runs with min and median does

    The spread on a shared machine is large.

  6. Which loop can a parallel do not handle?

    Answer

    One where an iteration reads what an earlier iteration wrote

    Independence of the iterations is the requirement.

  7. What does gfortran report for use omp_lib built without -fopenmp?

    Answer

    A link error: undefined reference to omp_get_max_threads_

    The module is shipped, the library is not linked.

  8. Why compare the integral of the two builds?

    Answer

    A different number means a race, and the timing would be worthless

    Correctness first, then the speed.

  9. Why is the serial sum(a) inside the timed region?

    Answer

    It is part of the work the user waits for

    A speedup must include everything that runs.

  10. What does the underscore in omp_get_max_threads_ come from?

    Answer

    The name mangling the Fortran compiler uses for an external procedure

    It is the symbol the compiler emits.

The programs on this page were built with GNU Fortran (Alpine 15.2.0) 15.2.0 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev

Banner image