Coarrays without OpenCoarrays
Coarrays are the one part of this course that this image can compile and cannot run in parallel, and the honest way to end the part is to show the exact limit. -fcoarray=lib turns a coarray program into one that calls a runtime, and the runtime is OpenCoarrays, which is not installed here. The build fails at the link step with the real message, and this page is about what that message means, what the library provides, and what a program can still do without it.
-fcoarray=lib
The flag has two useful values. single compiles the coarray syntax into one image's local memory, which is what the previous three lessons ran. lib compiles the same program into calls to a communication library: every remote reference becomes a function call, every sync all becomes a barrier, and every collective becomes a reduction routine. The compiler stops being self-contained at that point, and the program needs the library at link time and at run time.
The file caf.f90:
program exchange
implicit none
integer :: me, n
real :: x[*], total, biggest
me = this_image()
n = num_images()
x = real(me) ** 2
sync all
print '(a,i0,a,i0,a,f6.1)', 'image ', me, ' of ', n, ': x = ', x
total = x[1]
call co_sum(total)
print '(a,i0,a,f8.1)', 'image ', me, ': sum = ', total
biggest = x
call co_max(biggest)
print '(a,i0,a,f8.1)', 'image ', me, ': max = ', biggest
print '(a,f8.1)', 'four parts 1, 4, 9, 16 add to ', sum([1.0, 4.0, 9.0, 16.0])
x = x[1] + 100.0
sync all
print '(a,i0,a,f6.1)', 'image ', me, ': x after remote read and write = ', x
end program exchange
gfortran -Wall -Wextra -std=f2018 -fcoarray=single caf.f90 -o caf
./caf
image 1 of 1: x = 1.0
image 1: sum = 1.0
image 1: max = 1.0
four parts 1, 4, 9, 16 add to 30.0
image 1: x after remote read and write = 101.0
The same source with the other flag compiles and then stops at the link:
The link error
gfortran -Wall -Wextra -std=f2018 -fcoarray=lib caf.f90 -o caf_mpi
/usr/lib/gcc/x86_64-alpine-linux-musl/15.2.0/../../../../x86_64-alpine-linux-musl/bin/ld:
/tmp/ccbikfjj.o: in function `_caf_init.4.1':
caf.f90:(.text+0x1c): undefined reference to `_gfortran_caf_register_accessor'
caf.f90:(.text+0x30): undefined reference to `_gfortran_caf_register_accessor'
caf.f90:(.text+0x35): undefined reference to `_gfortran_caf_register_accessors_finish'
... sixteen more lines of the same kind ...
caf.f90:(.text+0x8bf): undefined reference to `_gfortran_caf_finalize'
collect2: error: ld returned 1 exit status
Every undefined name is a _gfortran_caf_* symbol: the compiler wrote the calls and the linker cannot find the code. The list reads like an inventory of the feature, which is useful in itself: _gfortran_caf_init and _gfortran_caf_finalize start and stop the runtime, _gfortran_caf_this_image and _gfortran_caf_num_images are the two enquiries, _gfortran_caf_sync_all is the barrier, _gfortran_caf_get_from_remote is a remote read, and _gfortran_caf_register sets up a coarray at start.
Twenty-two lines in total, of which the first four and the last two are shown. The middle of the list repeats the pattern with the names of the operations this program uses: co_sum, co_max, the remote read, the barrier.
What OpenCoarrays adds
OpenCoarrays is the library that provides those symbols, and it builds them on top of MPI. The package installs two things: libcaf_mpi.a (or the shared equivalent) and a launcher called cafrun. The link line gains one library, and the run line becomes cafrun -n 4 ./caf, which starts four processes and gives each one an image number.
This image has the single-image part of the runtime and not the parallel one. The directory that holds the compiler's libraries contains libcaf_single.a, which is what -fcoarray=single uses, and gfortran -print-file-name=libcaf_mpi.a echoes the name back without a path, which is how a missing library is reported. There is no cafrun on the PATH either.
What still works here
Everything the previous three lessons showed: the syntax, the local half of every algorithm, the barriers, the collectives with one image, the bounds and the image enquiries. What does not work is anything that needs two images: a race, a deadlock, a remote value that differs from the local one, and the actual speed of a communication.
That is a real limit of a course that has to prove every program it shows. A page could print a made-up run with four images and no reader could tell, and this course does not do that: it says which runs happened, and for coarrays that means one image, with the arithmetic of the larger case written out where it helps.
A program that ports
A coarray program written against the standard needs no change to run on one image, four or four thousand: it asks num_images() and divides the work. The build is the part that changes, and a project keeps it in one place: a makefile variable that is -fcoarray=single on a laptop and -fcoarray=lib -lcaf_mpi on the cluster, with cafrun in the run script.
The testing strategy follows the same line. A single-image run in the test suite catches the local bugs and the wrong statements; a four-image run on the cluster catches the synchronisation. A program that is only ever tested on one image is tested for half of its behaviour, which is exactly the half that does not need coarrays at all.
Exercises
- Build a coarray program with -fcoarray=lib and count the undefined symbols with 2>&1 | grep -c undefined. How many are there for the program above?
- Compile the same file with -fcoarray=none and read the first error. What does the flag mean?
- Write the link line that would be needed with libcaf_mpi.a installed, including the MPI libraries, and say which part of it you cannot test here.
- Take the program of lesson 098 and make the number of images part of the output of every line. Which lines would differ between one image and four?
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
What does
-fcoarray=libdo?Answer
Compiles the coarray operations into calls to a runtime library
The calls are _gfortran_caf_* and need OpenCoarrays.
-
What fails without OpenCoarrays?
Answer
The link, with undefined _gfortran_caf_ references
The compiler emits calls the linker cannot resolve.
-
Which library is present in this image?
Answer
libcaf_single.a
The single-image runtime is part of the compiler package.
-
What does
gfortran -print-file-name=libcaf_mpi.aprint when the library is missing?Answer
The name itself, without a path
It is the documented way to see that a library is absent.
-
How is a multi-image program started once OpenCoarrays is installed?
Answer
cafrun -n 4 ./program
cafrun is the launcher of OpenCoarrays; it is not here.
-
What can a single-image run test?
Answer
The local algorithm, the syntax and the statements, but not the races
Two images are what a synchronisation bug needs.
-
Which symbol sets up a coarray at program start?
Answer
_gfortran_caf_register
The link error names it, and it is the registration call.
-
What does a project put in one place?
Answer
The coarray build flags, single on a laptop and lib on a cluster
The build is the part that changes between the two.
-
Why does this course not show a four-image run?
Answer
Because the runtime is not in the image and an invented output would be a lie
Every output on a page is copied from a real run.
-
What does the linker error end with?
Answer
collect2: error: ld returned 1 exit status
It is the standard last line of a failed link.
The programs on this page were built with GNU Fortran (Alpine 15.2.0) 15.2.0 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev