Rate limiting and backpressure
A rate limit protects a service from its clients, and backpressure protects a process from its own queue. Go has no limiter in the standard library beyond the ticker and the buffered channel, and those two are enough to build the three shapes a service needs: a steady pace, a burst of tokens, and a queue that refuses work when it is full.
A ticker as a limiter
time.NewTicker sends a value on its channel at a fixed interval. Receiving from it before each request paces the loop: the first request waits one interval, the second waits for the next tick, and the spacing is the interval whether the work was fast or slow. The timestamps below are the real ones from the run.
package main
import (
"fmt"
"time"
)
func main() {
limiter := time.NewTicker(10 * time.Millisecond)
defer limiter.Stop()
start := time.Now()
for i := 1; i <= 3; i++ {
<-limiter.C
fmt.Printf("request %d at %v\n", i, time.Since(start).Round(time.Millisecond))
}
}
go build -o app main.go && ./app
request 1 at 10ms
request 2 at 21ms
request 3 at 30ms
The first tick came ten milliseconds after the ticker started, and the next two followed one interval apart; the extra millisecond on the second line is scheduling noise. A ticker is a good limiter for a worker that processes a queue at a fixed rate, and a poor one for a request path: every request waits for a tick even when the budget is unused.
A token bucket
A token bucket allows a burst and then a steady refill. The channel below holds the tokens: two are available at the start, a call takes one without blocking, and a call that finds the bucket empty is refused. The refill is a single send, which a ticker would do in a loop. This is the algorithm behind most API rate limiters, usually with a bucket per client.
package main
import (
"fmt"
"time"
)
func main() {
tokens := make(chan struct{}, 2)
tokens <- struct{}{}
tokens <- struct{}{}
allow := func() bool {
select {
case <-tokens:
return true
default:
return false
}
}
fmt.Println("call 1:", allow())
fmt.Println("call 2:", allow())
fmt.Println("call 3:", allow())
time.Sleep(10 * time.Millisecond)
tokens <- struct{}{}
fmt.Println("refilled, call 4:", allow())
}
go build -o app main.go && ./app
call 1: true
call 2: true
call 3: false
refilled, call 4: true
Two calls took the two tokens, the third found the bucket empty and returned false rather than waiting, and the refill allowed one more. A limiter that returns false lets the caller decide between a 429 and a delay; a limiter that blocks is a queue with a different name, and a queue in front of an overloaded service is how a small problem becomes a large one.
A bounded queue that drops
A buffered channel is a queue with a limit. The producer below tries five items against a queue of three and counts what was accepted and what was dropped: the select with a default is a non-blocking send, and the default is the load-shedding policy. A service that would rather answer 503 quickly than time out slowly makes the same choice in its request handler.
package main
import "fmt"
func main() {
queue := make(chan int, 3)
accepted, dropped := 0, 0
for i := 1; i <= 5; i++ {
select {
case queue <- i:
accepted++
default:
dropped++
}
}
fmt.Println("accepted:", accepted)
fmt.Println("dropped:", dropped)
fmt.Println("queued:", len(queue))
}
go build -o app main.go && ./app
accepted: 3
dropped: 2
queued: 3
Three items went in and two were refused, and the queue holds exactly its capacity. The same program with a blocking send would have waited for a consumer instead of dropping, which is backpressure rather than load shedding: both are valid, and a service should choose one on purpose. The size of the queue is the amount of work the process is willing to lose.
Backpressure
Backpressure is what an unbuffered channel does: the producer cannot run ahead of the consumer, so a slow consumer slows the producer. It is the right default between stages of one process, because the whole pipeline stays at the speed of its slowest part and the memory stays bounded. It is the wrong default at the edge of a service, where the producer is a client that cannot be slowed: there the queue either drops or grows, and growing is how a process runs out of memory.
A server therefore needs three numbers: how many requests it serves at once, how many it queues, and how long it lets a request wait. The limiter, the bounded queue and the context deadline are the three mechanisms, and a service that has all three degrades in a predictable way instead of falling over.
Exercises
- Change the ticker to a hundred milliseconds and run the program. What happens to the timestamps, and what would happen to a server that used this for every request?
- Fill the token bucket with ten tokens and make ten calls. Which calls are allowed, and what does the bucket model that a ticker does not?
- Replace the default in the bounded queue with a blocking send and add a consumer that sleeps. What does the producer do when the queue is full, and which policy would you choose for an HTTP handler?
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
What does time.NewTicker do?
Answer
Sends a value on its channel at a fixed interval
Receiving from the channel paces the loop.
-
What is a token bucket?
Answer
A budget of tokens that allows a burst and then a steady rate
A call takes a token or is refused.
-
What does a select with a default do for a full channel?
Answer
Drops the value and takes the default
It is a non-blocking send.
-
What is backpressure?
Answer
A slow consumer slowing the producer
It keeps memory bounded by slowing the source.
-
When is dropping better than blocking?
Answer
At the edge of a service where the client cannot be slowed
A fast 503 beats a slow timeout.
-
What does the size of a queue express?
Answer
How much work the process is willing to hold or lose
It is a memory and latency budget.
-
What does an unbuffered channel do to a pipeline?
Answer
Keeps each stage at the speed of the next one
The send waits for a receive.
-
Which three mechanisms bound a server?
Answer
Limiter, bounded queue and deadline
They bound rate, memory and time.
-
What did the ticker program print for the first request?
Answer
request 1 at 10ms
The first tick comes one interval after the start.
-
Why is a limiter that blocks a queue in disguise?
Answer
It makes callers wait, so the wait moves to the caller
The queue is just outside the process.
The code on this page was run with go 1.26.10 on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev