Coarray data exchange and reductions
A coarray program moves data through two mechanisms: a reference with a cosubscript, which reads or writes one image's copy from another, and the collective subroutines, which every image calls with its own value and which return a combined one to all of them. This lesson does both on one image, where the values are local, and follows the arithmetic of a four-image reduction so that a reader can see what the parallel case computes without a parallel run.
A remote read
x[1] reads the value of x on image one. After a sync all the read is guaranteed to see what image one wrote before its own barrier, and it costs whatever the runtime charges for a remote load: a message on a cluster, a cache line on a shared-memory node. A program that reads a remote value inside an inner loop pays that cost every iteration, which is why a coarray algorithm copies its neighbours into local memory once per step.
The file exch.f90:
program exchange
implicit none
integer :: me, n
real :: x[*], total, biggest
me = this_image()
n = num_images()
x = real(me) ** 2
sync all
print '(a,i0,a,i0,a,f6.1)', 'image ', me, ' of ', n, ': x = ', x
total = x[1]
call co_sum(total)
print '(a,i0,a,f8.1)', 'image ', me, ': sum = ', total
biggest = x
call co_max(biggest)
print '(a,i0,a,f8.1)', 'image ', me, ': max = ', biggest
print '(a,f8.1)', 'four parts 1, 4, 9, 16 add to ', sum([1.0, 4.0, 9.0, 16.0])
x = x[1] + 100.0
sync all
print '(a,i0,a,f6.1)', 'image ', me, ': x after remote read and write = ', x
end program exchange
gfortran -Wall -Wextra -std=f2018 -fcoarray=single exch.f90 -o exch
./exch
image 1 of 1: x = 1.0
image 1: sum = 1.0
image 1: max = 1.0
four parts 1, 4, 9, 16 add to 30.0
image 1: x after remote read and write = 101.0
A remote write
x[1] = value writes into another image's copy. It is a put: the statement completes when the local part is done and says nothing about when the other image sees the value, which is what the sync all after it is for. The last line of the run is the local half of that: on one image the read x[1] and the assignment are the same cell, so x becomes 101.
With four images the same statement on image two would write into image one and leave images two, three and four alone. A program that updates a halo of neighbours does exactly this, once per neighbour, and the cost of the barrier afterwards is why a coarray algorithm groups its updates into one phase rather than looping over neighbours with a sync each time.
CO_SUM
call co_sum(total) is a collective: every image calls it, each passes its own value, and every image receives the sum of all of them. It is not a coarray statement and does not need a cosubscript; the runtime knows which images take part. With four images whose values are 1, 4, 9 and 16 the call leaves 30 on every image, and the line above shows that arithmetic for the four-part array.
A collective is a synchronisation point in itself: no image returns before every image has called it, so the barriers around it are about ordering the data, not about the reduction. A program that puts a collective inside a branch taken by some images deadlocks, and it is the classic coarray mistake after the unpaired sync images of the previous lesson.
The other collectives
co_max and co_min do what their names say for a numeric argument and for a character one; co_all is a logical and; co_reduce takes a user function of two arguments and gives every image the result of folding it over the whole set, which is how a program combines a custom type. co_broadcast sends one image's value to all of them, and its second argument names the source image.
Each of them takes result_image= to deliver the answer to one image only, which saves the others the work of receiving it. That is the form a program uses when image one is the writer and the others only send, and it is a small but real saving on a large image count.
A reduction by hand
Before the collective subroutines arrived in Fortran 2018, a sum over images was a loop of remote reads and a local addition, and a program that needed a tree reduction wrote one. The modern call is shorter and lets the runtime use whatever the network offers, but the manual version is still what a program does when the operation is not one of the five: read the part from each image, combine it locally, and broadcast the result if everybody needs it.
The arithmetic is the same either way. A sum of four parts is three additions, a tree is two rounds of two, and the difference between them is latency rather than work. That is why a coarray reduction of a large array is worth doing and a coarray reduction of a scalar is worth checking: the collective has a fixed cost that a program with a hundred thousand tiny reductions will feel.
A character to CO_SUM
The numeric collectives take numeric arguments, and the message names the dummy as the standard does:
program badsum
implicit none
character(len=4) :: s
s = 'abcd'
call co_sum(s)
print *, s
end program badsum
gfortran -Wall -Wextra -std=f2018 -fcoarray=single badsum.f90 -o badsum
badsum.f90:6:14:
6 | call co_sum(s)
| 1
Error: 'a' argument of 'co_sum' intrinsic at (1) must have a numeric type
The dummy is called a in this intrinsic, which is why the message is so short. A character reduction is co_max or co_min, which do accept one.
One image
Every number on this page came from one image. A collective with one participant returns its own argument, so co_sum and co_max of 1.0 both give 1.0, and the remote references are local. The mechanism is exercised and the arithmetic is shown, and what is not shown is the part that needs the OpenCoarrays runtime: the next lesson builds with -fcoarray=lib and shows the real link error this image gives.
Exercises
- Set x = real(me) and print x[1] after a barrier. What does one image print, and what would four?
- Call co_min after co_max on the same variable. Is the second call computing anything new?
- Write the manual sum: a loop over i = 1, num_images() adding x[i]. Which barrier does it need before the loop?
- Reduce a character with co_max and with co_min and print both. Which one is a lexicographic maximum?
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
What does
x[1]read?Answer
The copy of x on image one
The cosubscript names the image.
-
What does
call co_sum(total)leave on every image?Answer
The sum of the values every image passed in
It is a collective; all images take part.
-
With four images holding 1, 4, 9 and 16, what does CO_MAX give every image?
Answer
16
The largest of the four values.
-
What is wrong with calling a collective inside a branch only some images take?
Answer
The others never arrive and the program deadlocks
A collective is a synchronisation point.
-
What does
result_image=do on a collective?Answer
Delivers the result to one image only
The others do not have to receive the answer.
-
What does a remote write promise?
Answer
That the value is sent, and nothing about when the other image sees it
A barrier or an atomic operation is what orders it.
-
Which collective combines a custom type?
Answer
co_reduce with a function of two arguments
It folds the user function over the set.
-
What does gfortran report for
call co_sum(s)with a character s?Answer
Error: the argument must have a numeric type
A character reduction is co_max or co_min.
-
What does a collective of one image return?
Answer
Its own argument
The reduction of one value is that value.
-
Why copy a neighbour's boundary into local memory instead of reading it every iteration?
Answer
A remote read costs more than a local one, so it is paid once per step
The cost of a remote load is per access.
The programs on this page were built with GNU Fortran (Alpine 15.2.0) 15.2.0 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev