Races, deadlocks and leaks
Concurrency fails in three ways. A data race is two accesses without an order, and the detector finds it. A deadlock is a cycle of waits, and the runtime fails the program when every goroutine is blocked. A leak is a goroutine that is blocked on something that will never happen, and nothing fails at all. This lesson shows one of each, with the output the runtime printed.
Two locks in opposite order
Main takes lock a, the worker takes b, and each then wants the other. The two channels make the interleaving certain: the worker signals that it holds b before main tries it, so main blocks on b while the worker blocks on a. Neither can move, and the runtime notices.
package main
import (
"fmt"
"sync"
)
func main() {
var a, b sync.Mutex
ready := make(chan struct{})
done := make(chan struct{})
go func() {
b.Lock()
defer b.Unlock()
close(ready)
a.Lock()
defer a.Unlock()
close(done)
}()
a.Lock()
defer a.Unlock()
<-ready
b.Lock()
defer b.Unlock()
<-done
fmt.Println("done")
}
go build -trimpath -o app main.go && ./app
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [sync.Mutex.Lock]:
internal/sync.runtime_SemacquireMutex(0x100a16e8a1e0?, 0x98?, 0x4530bf?)
runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0x100a16f1c028)
internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
sync/mutex.go:46
main.main()
./main.go:25 +0x14f
goroutine 19 [sync.Mutex.Lock]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0x100a16f1c020)
internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
sync/mutex.go:46
main.main.func1()
./main.go:17 +0xb2
created by main.main in goroutine 1
./main.go:13 +0xd8
exit status: 2
The dump names the lock each goroutine waits on and the line it called from: main at line 25 waits for b, the worker at line 17 waits for a. The fix is an order: every goroutine takes the same two locks in the same sequence, or the two fields belong to one struct with one lock. A deadlock that involves a channel instead of a mutex gets the same fatal error with a different state line.
A goroutine that never ends
The leak function starts a goroutine that sends on a channel nobody will ever receive from. The goroutine stays blocked on the send, the channel keeps it alive, and nothing reports the problem. runtime.NumGoroutine is the cheap way to see it: ten calls left ten goroutines behind.
package main
import (
"fmt"
"runtime"
"time"
)
func leak() {
ch := make(chan int)
go func() {
ch <- 1
}()
}
func main() {
before := runtime.NumGoroutine()
for i := 0; i < 10; i++ {
leak()
}
time.Sleep(10 * time.Millisecond)
after := runtime.NumGoroutine()
fmt.Println("before:", before)
fmt.Println("after:", after)
}
go build -o app main.go && ./app
before: 1
after: 11
A program that ends frees them all, which is why the leak is invisible here. In a server the same code runs once per request and the count climbs until the process is restarted. The fix is a way out of the send: a buffered channel of size one, a select with ctx.Done(), or a receiver that always drains the channel. A goroutine needs to know how it is allowed to end.
A panic in a goroutine
A panic in any goroutine ends the whole program, whether or not it was the one that started it. There is no per-goroutine recovery: the stack unwinds through the panicking goroutine only, and if nothing there recovers, the runtime prints the stack and exits. The main goroutine below is waiting on an empty select when the worker fails, and the process still dies.
package main
import "fmt"
func main() {
go func() {
panic("worker failed")
}()
fmt.Println("main is waiting")
select {}
}
go build -trimpath -o app main.go && ./app
main is waiting
panic: worker failed
goroutine 19 [running]:
main.main.func1()
./main.go:7 +0x25
created by main.main in goroutine 1
./main.go:6 +0x1a
exit status: 2
The trace says which goroutine panicked, the line of the panic and the line that created it. This is why a server recovers in its worker goroutine and logs the stack: a panic in one request must not take the process with it, and recover belongs at the boundary of the goroutine where the work was started.
Finding the three
A data race needs go test -race or go build -race, and the detector reports both accesses with their stacks. A deadlock announces itself with the fatal error and the dump above, and GOTRACEBACK=all adds the goroutines that were not in the cycle. A leak has no report at all, so a test measures it: record runtime.NumGoroutine before, run the code, wait briefly, and compare. A test that leaks a goroutine per case grows a package suite until it times out.
A profile shows where the time goes when the symptom is a slowdown rather than a failure: a goroutine dump of a stuck server is one HTTP request to /debug/pprof/goroutine?debug=2 when net/http/pprof is imported, and it lists every goroutine and the line it waits on. That page is the first place to look when a process stops serving and does not crash.
Exercises
- Fix the deadlock by changing the order of the two locks in the worker and run the program. Why does the same fix work for any number of goroutines?
- Add a receiver for the leaked channel and print the count again. Which line changes, and what does the goroutine do after the send?
- Add a recover in the panicking goroutine and print the recovered value. What does the program do after the recover, and what happens to main?
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
What does the runtime print when every goroutine is blocked?
Answer
fatal error: all goroutines are asleep - deadlock!
The program exits with status 2.
-
How is a deadlock between two mutexes avoided?
Answer
By taking the locks in the same order everywhere
A global order makes a cycle impossible.
-
What does the dump name for each blocked goroutine?
Answer
The state and the line it waits on
The stack shows the call that blocks.
-
What is a goroutine leak?
Answer
A goroutine blocked on something that never happens
It stays alive and does nothing.
-
Which call counts the goroutines?
Answer
runtime.NumGoroutine
It is the cheap leak check in a test.
-
What ends a goroutine blocked on a send nobody receives?
Answer
A select on ctx.Done, a buffer or a receiver
It needs a way out of the send.
-
What happens when a goroutine panics without recover?
Answer
The whole program exits
The runtime prints the stack and exits with status 2.
-
Where does recover belong in a server?
Answer
At the boundary of the goroutine that runs the request
One request must not take the process down.
-
Which tool finds a data race?
Answer
The race detector with -race
It instruments the memory accesses.
-
Which profile lists every goroutine and where it waits?
Answer
The goroutine profile
debug=2 prints the full stacks.
The code on this page was run with go 1.26.10 on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev