Coarray data exchange and reductions

A coarray program moves data through two mechanisms: a reference with a cosubscript, which reads or writes one image's copy from another, and the collective subroutines, which every image calls with its own value and which return a combined one to all of them. This lesson does both on one image, where the values are local, and follows the arithmetic of a four-image reduction so that a reader can see what the parallel case computes without a parallel run.

A remote read

x[1] reads the value of x on image one. After a sync all the read is guaranteed to see what image one wrote before its own barrier, and it costs whatever the runtime charges for a remote load: a message on a cluster, a cache line on a shared-memory node. A program that reads a remote value inside an inner loop pays that cost every iteration, which is why a coarray algorithm copies its neighbours into local memory once per step.

The file exch.f90:

program exchange implicit none integer :: me, n real :: x[*], total, biggest me = this_image() n = num_images() x = real(me) ** 2 sync all print '(a,i0,a,i0,a,f6.1)', 'image ', me, ' of ', n, ': x = ', x total = x[1] call co_sum(total) print '(a,i0,a,f8.1)', 'image ', me, ': sum = ', total biggest = x call co_max(biggest) print '(a,i0,a,f8.1)', 'image ', me, ': max = ', biggest print '(a,f8.1)', 'four parts 1, 4, 9, 16 add to ', sum([1.0, 4.0, 9.0, 16.0]) x = x[1] + 100.0 sync all print '(a,i0,a,f6.1)', 'image ', me, ': x after remote read and write = ', x end program exchange

gfortran -Wall -Wextra -std=f2018 -fcoarray=single exch.f90 -o exch
./exch

image 1 of 1: x = 1.0
image 1: sum = 1.0
image 1: max = 1.0
four parts 1, 4, 9, 16 add to 30.0
image 1: x after remote read and write = 101.0

A remote write

x[1] = value writes into another image's copy. It is a put: the statement completes when the local part is done and says nothing about when the other image sees the value, which is what the sync all after it is for. The last line of the run is the local half of that: on one image the read x[1] and the assignment are the same cell, so x becomes 101.

With four images the same statement on image two would write into image one and leave images two, three and four alone. A program that updates a halo of neighbours does exactly this, once per neighbour, and the cost of the barrier afterwards is why a coarray algorithm groups its updates into one phase rather than looping over neighbours with a sync each time.

CO_SUM

call co_sum(total) is a collective: every image calls it, each passes its own value, and every image receives the sum of all of them. It is not a coarray statement and does not need a cosubscript; the runtime knows which images take part. With four images whose values are 1, 4, 9 and 16 the call leaves 30 on every image, and the line above shows that arithmetic for the four-part array.

A collective is a synchronisation point in itself: no image returns before every image has called it, so the barriers around it are about ordering the data, not about the reduction. A program that puts a collective inside a branch taken by some images deadlocks, and it is the classic coarray mistake after the unpaired sync images of the previous lesson.

The other collectives

co_max and co_min do what their names say for a numeric argument and for a character one; co_all is a logical and; co_reduce takes a user function of two arguments and gives every image the result of folding it over the whole set, which is how a program combines a custom type. co_broadcast sends one image's value to all of them, and its second argument names the source image.

Each of them takes result_image= to deliver the answer to one image only, which saves the others the work of receiving it. That is the form a program uses when image one is the writer and the others only send, and it is a small but real saving on a large image count.

A reduction by hand

Before the collective subroutines arrived in Fortran 2018, a sum over images was a loop of remote reads and a local addition, and a program that needed a tree reduction wrote one. The modern call is shorter and lets the runtime use whatever the network offers, but the manual version is still what a program does when the operation is not one of the five: read the part from each image, combine it locally, and broadcast the result if everybody needs it.

The arithmetic is the same either way. A sum of four parts is three additions, a tree is two rounds of two, and the difference between them is latency rather than work. That is why a coarray reduction of a large array is worth doing and a coarray reduction of a scalar is worth checking: the collective has a fixed cost that a program with a hundred thousand tiny reductions will feel.

A character to CO_SUM

The numeric collectives take numeric arguments, and the message names the dummy as the standard does:

program badsum implicit none character(len=4) :: s s = 'abcd' call co_sum(s) print *, s end program badsum

gfortran -Wall -Wextra -std=f2018 -fcoarray=single badsum.f90 -o badsum

badsum.f90:6:14:

6 | call co_sum(s)
| 1
Error: 'a' argument of 'co_sum' intrinsic at (1) must have a numeric type

The dummy is called a in this intrinsic, which is why the message is so short. A character reduction is co_max or co_min, which do accept one.

One image

Every number on this page came from one image. A collective with one participant returns its own argument, so co_sum and co_max of 1.0 both give 1.0, and the remote references are local. The mechanism is exercised and the arithmetic is shown, and what is not shown is the part that needs the OpenCoarrays runtime: the next lesson builds with -fcoarray=lib and shows the real link error this image gives.

Exercises

  1. Set x = real(me) and print x[1] after a barrier. What does one image print, and what would four?
  2. Call co_min after co_max on the same variable. Is the second call computing anything new?
  3. Write the manual sum: a loop over i = 1, num_images() adding x[i]. Which barrier does it need before the loop?
  4. Reduce a character with co_max and with co_min and print both. Which one is a lexicographic maximum?

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. What does x[1] read?

    Answer

    The copy of x on image one

    The cosubscript names the image.

  2. What does call co_sum(total) leave on every image?

    Answer

    The sum of the values every image passed in

    It is a collective; all images take part.

  3. With four images holding 1, 4, 9 and 16, what does CO_MAX give every image?

    Answer

    16

    The largest of the four values.

  4. What is wrong with calling a collective inside a branch only some images take?

    Answer

    The others never arrive and the program deadlocks

    A collective is a synchronisation point.

  5. What does result_image= do on a collective?

    Answer

    Delivers the result to one image only

    The others do not have to receive the answer.

  6. What does a remote write promise?

    Answer

    That the value is sent, and nothing about when the other image sees it

    A barrier or an atomic operation is what orders it.

  7. Which collective combines a custom type?

    Answer

    co_reduce with a function of two arguments

    It folds the user function over the set.

  8. What does gfortran report for call co_sum(s) with a character s?

    Answer

    Error: the argument must have a numeric type

    A character reduction is co_max or co_min.

  9. What does a collective of one image return?

    Answer

    Its own argument

    The reduction of one value is that value.

  10. Why copy a neighbour's boundary into local memory instead of reading it every iteration?

    Answer

    A remote read costs more than a local one, so it is paid once per step

    The cost of a remote load is per access.

The programs on this page were built with GNU Fortran (Alpine 15.2.0) 15.2.0 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev