Coarrays without OpenCoarrays

Coarrays are the one part of this course that this image can compile and cannot run in parallel, and the honest way to end the part is to show the exact limit. -fcoarray=lib turns a coarray program into one that calls a runtime, and the runtime is OpenCoarrays, which is not installed here. The build fails at the link step with the real message, and this page is about what that message means, what the library provides, and what a program can still do without it.

-fcoarray=lib

The flag has two useful values. single compiles the coarray syntax into one image's local memory, which is what the previous three lessons ran. lib compiles the same program into calls to a communication library: every remote reference becomes a function call, every sync all becomes a barrier, and every collective becomes a reduction routine. The compiler stops being self-contained at that point, and the program needs the library at link time and at run time.

The file caf.f90:

program exchange implicit none integer :: me, n real :: x[*], total, biggest me = this_image() n = num_images() x = real(me) ** 2 sync all print '(a,i0,a,i0,a,f6.1)', 'image ', me, ' of ', n, ': x = ', x total = x[1] call co_sum(total) print '(a,i0,a,f8.1)', 'image ', me, ': sum = ', total biggest = x call co_max(biggest) print '(a,i0,a,f8.1)', 'image ', me, ': max = ', biggest print '(a,f8.1)', 'four parts 1, 4, 9, 16 add to ', sum([1.0, 4.0, 9.0, 16.0]) x = x[1] + 100.0 sync all print '(a,i0,a,f6.1)', 'image ', me, ': x after remote read and write = ', x end program exchange

gfortran -Wall -Wextra -std=f2018 -fcoarray=single caf.f90 -o caf
./caf

image 1 of 1: x = 1.0
image 1: sum = 1.0
image 1: max = 1.0
four parts 1, 4, 9, 16 add to 30.0
image 1: x after remote read and write = 101.0

The same source with the other flag compiles and then stops at the link:

The link error

gfortran -Wall -Wextra -std=f2018 -fcoarray=lib caf.f90 -o caf_mpi

/usr/lib/gcc/x86_64-alpine-linux-musl/15.2.0/../../../../x86_64-alpine-linux-musl/bin/ld: /tmp/ccbikfjj.o: in function `_caf_init.4.1':
caf.f90:(.text+0x1c): undefined reference to `_gfortran_caf_register_accessor'
caf.f90:(.text+0x30): undefined reference to `_gfortran_caf_register_accessor'
caf.f90:(.text+0x35): undefined reference to `_gfortran_caf_register_accessors_finish'
... sixteen more lines of the same kind ...
caf.f90:(.text+0x8bf): undefined reference to `_gfortran_caf_finalize'
collect2: error: ld returned 1 exit status

Every undefined name is a _gfortran_caf_* symbol: the compiler wrote the calls and the linker cannot find the code. The list reads like an inventory of the feature, which is useful in itself: _gfortran_caf_init and _gfortran_caf_finalize start and stop the runtime, _gfortran_caf_this_image and _gfortran_caf_num_images are the two enquiries, _gfortran_caf_sync_all is the barrier, _gfortran_caf_get_from_remote is a remote read, and _gfortran_caf_register sets up a coarray at start.

Twenty-two lines in total, of which the first four and the last two are shown. The middle of the list repeats the pattern with the names of the operations this program uses: co_sum, co_max, the remote read, the barrier.

What OpenCoarrays adds

OpenCoarrays is the library that provides those symbols, and it builds them on top of MPI. The package installs two things: libcaf_mpi.a (or the shared equivalent) and a launcher called cafrun. The link line gains one library, and the run line becomes cafrun -n 4 ./caf, which starts four processes and gives each one an image number.

This image has the single-image part of the runtime and not the parallel one. The directory that holds the compiler's libraries contains libcaf_single.a, which is what -fcoarray=single uses, and gfortran -print-file-name=libcaf_mpi.a echoes the name back without a path, which is how a missing library is reported. There is no cafrun on the PATH either.

What still works here

Everything the previous three lessons showed: the syntax, the local half of every algorithm, the barriers, the collectives with one image, the bounds and the image enquiries. What does not work is anything that needs two images: a race, a deadlock, a remote value that differs from the local one, and the actual speed of a communication.

That is a real limit of a course that has to prove every program it shows. A page could print a made-up run with four images and no reader could tell, and this course does not do that: it says which runs happened, and for coarrays that means one image, with the arithmetic of the larger case written out where it helps.

A program that ports

A coarray program written against the standard needs no change to run on one image, four or four thousand: it asks num_images() and divides the work. The build is the part that changes, and a project keeps it in one place: a makefile variable that is -fcoarray=single on a laptop and -fcoarray=lib -lcaf_mpi on the cluster, with cafrun in the run script.

The testing strategy follows the same line. A single-image run in the test suite catches the local bugs and the wrong statements; a four-image run on the cluster catches the synchronisation. A program that is only ever tested on one image is tested for half of its behaviour, which is exactly the half that does not need coarrays at all.

Exercises

  1. Build a coarray program with -fcoarray=lib and count the undefined symbols with 2>&1 | grep -c undefined. How many are there for the program above?
  2. Compile the same file with -fcoarray=none and read the first error. What does the flag mean?
  3. Write the link line that would be needed with libcaf_mpi.a installed, including the MPI libraries, and say which part of it you cannot test here.
  4. Take the program of lesson 098 and make the number of images part of the output of every line. Which lines would differ between one image and four?

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. What does -fcoarray=lib do?

    Answer

    Compiles the coarray operations into calls to a runtime library

    The calls are _gfortran_caf_* and need OpenCoarrays.

  2. What fails without OpenCoarrays?

    Answer

    The link, with undefined _gfortran_caf_ references

    The compiler emits calls the linker cannot resolve.

  3. Which library is present in this image?

    Answer

    libcaf_single.a

    The single-image runtime is part of the compiler package.

  4. What does gfortran -print-file-name=libcaf_mpi.a print when the library is missing?

    Answer

    The name itself, without a path

    It is the documented way to see that a library is absent.

  5. How is a multi-image program started once OpenCoarrays is installed?

    Answer

    cafrun -n 4 ./program

    cafrun is the launcher of OpenCoarrays; it is not here.

  6. What can a single-image run test?

    Answer

    The local algorithm, the syntax and the statements, but not the races

    Two images are what a synchronisation bug needs.

  7. Which symbol sets up a coarray at program start?

    Answer

    _gfortran_caf_register

    The link error names it, and it is the registration call.

  8. What does a project put in one place?

    Answer

    The coarray build flags, single on a laptop and lib on a cluster

    The build is the part that changes between the two.

  9. Why does this course not show a four-image run?

    Answer

    Because the runtime is not in the image and an invented output would be a lie

    Every output on a page is copied from a real run.

  10. What does the linker error end with?

    Answer

    collect2: error: ld returned 1 exit status

    It is the standard last line of a failed link.

The programs on this page were built with GNU Fortran (Alpine 15.2.0) 15.2.0 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev