Pipelines

A pipeline is a chain of goroutines connected by channels: each stage reads from its input, transforms, and writes to its output. Each stage is a function that takes a channel and returns a channel, starts one goroutine, and closes its output when its input ends. The shape composes, and the composition is the point.

A stage is a function

generate turns a list into a channel and square turns a channel of numbers into a channel of squares. Each closes its own output with a defer, so the consumer's range ends when the last value passed. The main function is the consumer and adds the values up as they arrive.

package main import "fmt" func generate(numbers ...int) <-chan int { out := make(chan int) go func() { defer close(out) for _, n := range numbers { out <- n } }() return out } func square(in <-chan int) <-chan int { out := make(chan int) go func() { defer close(out) for n := range in { out <- n * n } }() return out } func main() { sum := 0 for v := range square(generate(1, 2, 3, 4)) { fmt.Println("value:", v) sum += v } fmt.Println("sum:", sum) }

go build -o app main.go && ./app

value: 1
value: 4
value: 9
value: 16
sum: 30

Each stage holds one value at a time, so the pipeline uses no buffer memory and the values flow as fast as the slowest stage allows. A stage that is slower than the rest is where a buffer or another worker helps; the next lesson fans out exactly there.

Stopping early

The take stage passes three values and returns. The consumer finishes, but the generator is still blocked on a send nobody will receive, so it stays alive until the program ends. For one short program that is invisible; in a server it is a goroutine leak per request, and the fix is a done channel or a context that every stage watches.

package main import "fmt" func generate(n int) <-chan int { out := make(chan int) go func() { defer close(out) for i := 1; i <= n; i++ { out <- i } }() return out } func take(in <-chan int, n int) <-chan int { out := make(chan int) go func() { defer close(out) for v := range in { if n == 0 { return } out <- v n-- } }() return out } func main() { for v := range take(generate(10), 3) { fmt.Println("value:", v) } }

go build -o app main.go && ./app

value: 1
value: 2
value: 3

The output is right and the program ends, which is exactly why the leak is easy to miss. A fixed version passes a context to the generator and selects on ctx.Done() in every send, as the next section shows; the stage that stops early then cancels the context and the upstream goroutines return.

Cancelling the stages

The generator below selects between the send and the context, so a cancelled context ends the goroutine even when nobody is receiving. The consumer cancels after three values and stops reading; the count is the proof, and the goroutine returns on its next check instead of blocking forever.

package main import ( "context" "fmt" ) func generate(ctx context.Context, n int) <-chan int { out := make(chan int) go func() { defer close(out) for i := 1; i <= n; i++ { select { case out <- i: case <-ctx.Done(): return } } }() return out } func main() { ctx, cancel := context.WithCancel(context.Background()) defer cancel() count := 0 for v := range generate(ctx, 100) { count++ if count == 3 { cancel() break } _ = v } fmt.Println("received:", count) }

go build -o app main.go && ./app

received: 3

The generator could send once more between the cancel and its next select, and the consumer would receive it; the break makes the count three regardless. That is the honest contract of a pipeline: cancellation stops the work, not the value already in flight, and a caller that needs an exact cut applies it on its own side.

An iterator instead of a channel

Since Go 1.23 a function with the signature func(yield func(T) bool) can be ranged over directly. The iterator below produces the squares with a loop and no goroutine, no channel and no leak, and the consumer's break stops it by making yield return false. For a pipeline that transforms a sequence on one goroutine, this is simpler than a chain of channels.

package main import "fmt" func squares(n int) func(yield func(int) bool) { return func(yield func(int) bool) { for i := 1; i <= n; i++ { if !yield(i * i) { return } } } } func main() { for v := range squares(4) { fmt.Println("value:", v) } }

go build -o app main.go && ./app

value: 1
value: 4
value: 9
value: 16

The rule of thumb: channels when the stages run concurrently or cross a goroutine boundary, iterators when the work is a sequence consumed in one place. A channel pipeline buys parallelism and pays for it with goroutines that must be cancelled; an iterator buys simplicity and gives up parallelism.

Exercises

  1. Add a stage between generate and square that keeps only odd numbers. Which function's signature has to change, and which does not?
  2. Measure the time of the first pipeline with a time.Sleep in the square stage and with a buffer of ten on each channel. Which change helps, and why?
  3. Rewrite take as an iterator and remove the generator's goroutine. How many goroutines does the program have left?

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. What is a pipeline stage?

    Answer

    A function that takes a channel and returns a channel

    Each stage closes its own output.

  2. Who closes the output of a stage?

    Answer

    The stage itself, when its input ends

    A defer close in the stage goroutine.

  3. What happens when a stage returns early and the producer keeps sending?

    Answer

    The producer blocks forever and leaks

    Nobody receives, so the send never completes.

  4. What is the fix for a leaked producer?

    Answer

    A done channel or a context the producer watches

    The producer must be able to stop.

  5. What does a send in a select with ctx.Done() allow?

    Answer

    The send to be cancelled

    The send is abandoned when the context ends.

  6. What does the iterator signature func(yield func(int) bool) allow?

    Answer

    Range over the function as a sequence

    The consumer breaks by returning false from yield.

  7. When is a channel pipeline better than an iterator?

    Answer

    When the stages run concurrently

    The channel buys parallelism across goroutines.

  8. What does a buffer in a pipeline change?

    Answer

    How far a stage can run ahead

    It trades memory for slack.

  9. What does the sum pipeline print?

    Answer

    30

    1 + 4 + 9 + 16 is 30.

  10. What is the main risk of a long channel pipeline?

    Answer

    Leaked goroutines when a stage stops early

    Every goroutine needs a way to be cancelled.

The code on this page was run with go 1.26.10 on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev