Pipelines
A pipeline is a chain of goroutines connected by channels: each stage reads from its input, transforms, and writes to its output. Each stage is a function that takes a channel and returns a channel, starts one goroutine, and closes its output when its input ends. The shape composes, and the composition is the point.
A stage is a function
generate turns a list into a channel and square turns a channel of numbers into a channel of squares. Each closes its own output with a defer, so the consumer's range ends when the last value passed. The main function is the consumer and adds the values up as they arrive.
package main
import "fmt"
func generate(numbers ...int) <-chan int {
out := make(chan int)
go func() {
defer close(out)
for _, n := range numbers {
out <- n
}
}()
return out
}
func square(in <-chan int) <-chan int {
out := make(chan int)
go func() {
defer close(out)
for n := range in {
out <- n * n
}
}()
return out
}
func main() {
sum := 0
for v := range square(generate(1, 2, 3, 4)) {
fmt.Println("value:", v)
sum += v
}
fmt.Println("sum:", sum)
}
go build -o app main.go && ./app
value: 1
value: 4
value: 9
value: 16
sum: 30
Each stage holds one value at a time, so the pipeline uses no buffer memory and the values flow as fast as the slowest stage allows. A stage that is slower than the rest is where a buffer or another worker helps; the next lesson fans out exactly there.
Stopping early
The take stage passes three values and returns. The consumer finishes, but the generator is still blocked on a send nobody will receive, so it stays alive until the program ends. For one short program that is invisible; in a server it is a goroutine leak per request, and the fix is a done channel or a context that every stage watches.
package main
import "fmt"
func generate(n int) <-chan int {
out := make(chan int)
go func() {
defer close(out)
for i := 1; i <= n; i++ {
out <- i
}
}()
return out
}
func take(in <-chan int, n int) <-chan int {
out := make(chan int)
go func() {
defer close(out)
for v := range in {
if n == 0 {
return
}
out <- v
n--
}
}()
return out
}
func main() {
for v := range take(generate(10), 3) {
fmt.Println("value:", v)
}
}
go build -o app main.go && ./app
value: 1
value: 2
value: 3
The output is right and the program ends, which is exactly why the leak is easy to miss. A fixed version passes a context to the generator and selects on ctx.Done() in every send, as the next section shows; the stage that stops early then cancels the context and the upstream goroutines return.
Cancelling the stages
The generator below selects between the send and the context, so a cancelled context ends the goroutine even when nobody is receiving. The consumer cancels after three values and stops reading; the count is the proof, and the goroutine returns on its next check instead of blocking forever.
package main
import (
"context"
"fmt"
)
func generate(ctx context.Context, n int) <-chan int {
out := make(chan int)
go func() {
defer close(out)
for i := 1; i <= n; i++ {
select {
case out <- i:
case <-ctx.Done():
return
}
}
}()
return out
}
func main() {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
count := 0
for v := range generate(ctx, 100) {
count++
if count == 3 {
cancel()
break
}
_ = v
}
fmt.Println("received:", count)
}
go build -o app main.go && ./app
received: 3
The generator could send once more between the cancel and its next select, and the consumer would receive it; the break makes the count three regardless. That is the honest contract of a pipeline: cancellation stops the work, not the value already in flight, and a caller that needs an exact cut applies it on its own side.
An iterator instead of a channel
Since Go 1.23 a function with the signature func(yield func(T) bool) can be ranged over directly. The iterator below produces the squares with a loop and no goroutine, no channel and no leak, and the consumer's break stops it by making yield return false. For a pipeline that transforms a sequence on one goroutine, this is simpler than a chain of channels.
package main
import "fmt"
func squares(n int) func(yield func(int) bool) {
return func(yield func(int) bool) {
for i := 1; i <= n; i++ {
if !yield(i * i) {
return
}
}
}
}
func main() {
for v := range squares(4) {
fmt.Println("value:", v)
}
}
go build -o app main.go && ./app
value: 1
value: 4
value: 9
value: 16
The rule of thumb: channels when the stages run concurrently or cross a goroutine boundary, iterators when the work is a sequence consumed in one place. A channel pipeline buys parallelism and pays for it with goroutines that must be cancelled; an iterator buys simplicity and gives up parallelism.
Exercises
- Add a stage between generate and square that keeps only odd numbers. Which function's signature has to change, and which does not?
- Measure the time of the first pipeline with a time.Sleep in the square stage and with a buffer of ten on each channel. Which change helps, and why?
- Rewrite take as an iterator and remove the generator's goroutine. How many goroutines does the program have left?
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
What is a pipeline stage?
Answer
A function that takes a channel and returns a channel
Each stage closes its own output.
-
Who closes the output of a stage?
Answer
The stage itself, when its input ends
A defer close in the stage goroutine.
-
What happens when a stage returns early and the producer keeps sending?
Answer
The producer blocks forever and leaks
Nobody receives, so the send never completes.
-
What is the fix for a leaked producer?
Answer
A done channel or a context the producer watches
The producer must be able to stop.
-
What does a send in a select with ctx.Done() allow?
Answer
The send to be cancelled
The send is abandoned when the context ends.
-
What does the iterator signature func(yield func(int) bool) allow?
Answer
Range over the function as a sequence
The consumer breaks by returning false from yield.
-
When is a channel pipeline better than an iterator?
Answer
When the stages run concurrently
The channel buys parallelism across goroutines.
-
What does a buffer in a pipeline change?
Answer
How far a stage can run ahead
It trades memory for slack.
-
What does the sum pipeline print?
Answer
30
1 + 4 + 9 + 16 is 30.
-
What is the main risk of a long channel pipeline?
Answer
Leaked goroutines when a stage stops early
Every goroutine needs a way to be cancelled.
The code on this page was run with go 1.26.10 on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev