Runtime metrics and traces
A Go process already knows most of what a dashboard wants: how many goroutines run, how much the heap holds, how often the collector ran, how long a pause was. runtime/metrics exposes them by name with a stable path, runtime/trace records what the scheduler did, and net/http/pprof serves the profiles over HTTP. This lesson reads each one from a real run.
runtime/metrics
A metric has a name such as /sched/goroutines:goroutines and a value whose kind is part of the name after the colon. The program describes the samples it wants, calls Read once and prints them. The numbers below are this run; the names are the stable part.
package main
import (
"fmt"
"runtime/metrics"
)
func main() {
samples := []metrics.Sample{
{Name: "/sched/goroutines:goroutines"},
{Name: "/gc/heap/allocs:bytes"},
{Name: "/gc/cycles/total:gc-cycles"},
}
metrics.Read(samples)
for _, s := range samples {
fmt.Printf("%s = %v\n", s.Name, s.Value.Uint64())
}
}
go build -o app main.go && ./app
/sched/goroutines:goroutines = 6
/gc/heap/allocs:bytes = 57216
/gc/cycles/total:gc-cycles = 0
Six goroutines because the metrics package starts a few of its own, and no collection yet because the program is small. A service records these names on an interval and exports them to Prometheus with the same names, which is why the paths are versioned and documented. metrics.All lists every name a build supports, which is how a program discovers one it did not know.
MemStats
The older API is a struct with every counter. It is still the one most dashboards use and the one a program reads when it wants the heap and the collector in one call. ReadMemStats stops the world briefly, so it belongs on a timer and not in a handler.
package main
import (
"fmt"
"runtime"
)
func main() {
var stats runtime.MemStats
runtime.ReadMemStats(&stats)
fmt.Println("cpus:", runtime.NumCPU())
fmt.Println("goroutines:", runtime.NumGoroutine())
fmt.Println("heap alloc > 0:", stats.Alloc > 0)
fmt.Println("total alloc > 0:", stats.TotalAlloc > 0)
}
go build -o app main.go && ./app
cpus: 4
goroutines: 1
heap alloc > 0: true
total alloc > 0: true
A program at rest has one goroutine and a small heap. The booleans are there because the exact numbers change with every build; the counters that matter in production are Alloc, HeapInuse, NumGC and the pause percentiles, and a service exports them with its own labels.
A runtime trace
A trace records what the scheduler, the garbage collector and the goroutines did, with timestamps. The program starts a trace into a temporary file, runs three goroutines that sum a thousand numbers, stops the trace and checks that the file has bytes in it. The file is read with go tool trace.
package main
import (
"fmt"
"os"
"runtime/trace"
"sync"
)
func main() {
f, err := os.CreateTemp("", "trace-*.out")
if err != nil {
panic(err)
}
defer os.Remove(f.Name())
defer f.Close()
if err := trace.Start(f); err != nil {
panic(err)
}
var wg sync.WaitGroup
for i := 0; i < 3; i++ {
wg.Add(1)
go func() {
defer wg.Done()
sum := 0
for j := 0; j < 1000; j++ {
sum += j
}
_ = sum
}()
}
wg.Wait()
trace.Stop()
info, err := f.Stat()
if err != nil {
panic(err)
}
fmt.Println("trace bytes > 0:", info.Size() > 0)
}
go build -o app main.go && ./app
trace bytes > 0: true
The file is small and real, and go tool trace trace.out opens a page that shows the goroutines over time, the GC cycles and the scheduler latency. A trace is the tool for a latency question that a profile cannot answer: not where the CPU went, but what was runnable and did not run.
The pprof endpoints
Importing net/http/pprof registers handlers on http.DefaultServeMux: the index, the goroutine dump, the heap profile, the CPU profile and the trace. The program serves that mux and asks for the index, which lists the profiles and mentions goroutine among them.
package main
import (
"fmt"
"io"
"net/http"
"net/http/httptest"
"strings"
_ "net/http/pprof"
)
func main() {
srv := httptest.NewServer(http.DefaultServeMux)
defer srv.Close()
resp, err := http.Get(srv.URL + "/debug/pprof/")
if err != nil {
panic(err)
}
body, _ := io.ReadAll(resp.Body)
resp.Body.Close()
fmt.Println("status:", resp.Status)
fmt.Println("mentions goroutine:", strings.Contains(string(body), "goroutine"))
}
go build -o app main.go && ./app
status: 200 OK
mentions goroutine: true
The blank import is the switch, and a service mounts the handlers on a private port rather than on the public mux, because a heap profile and a goroutine dump are not for the internet. The path /debug/pprof/goroutine?debug=2 prints every goroutine and the line it waits on, which is the first thing to read when a server stops answering.
What to watch
Five numbers cover a Go service: goroutines over time, heap in use, GC cycles per second, the pause percentiles and the number of open connections. A goroutine count that climbs is a leak. A heap that climbs without a limit is a leak or a cache without one. A pause in the milliseconds is normal, and a pause in the hundreds is a signal to look. Everything else is a detail of the workload.
The lesson after the next one ends Part 9; the capstone project then uses these tools on a parser, with a benchmark and a profile of its own.
Exercises
- Add /sched/latencies:seconds to the samples and print its kind. Which accessor reads it, and why is it not a Uint64?
- Run the trace program with ten goroutines and open the file with go tool trace. What does the timeline show that the printed line does not?
- Serve the pprof mux on a listener and fetch /debug/pprof/goroutine?debug=2. How many goroutines does the dump list, and which one is the HTTP handler?
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
What is the name of a runtime metric?
Answer
A path with the kind after a colon
The path is stable across versions.
-
How is a metric value read?
Answer
By the accessor of its kind, such as Uint64
The kind is part of the name.
-
Why is ReadMemStats not for a handler?
Answer
It stops the world briefly
A timer is the place for it.
-
Which counters matter most in a service?
Answer
Goroutines, heap, GC cycles and pause percentiles
They show leaks and pressure.
-
What does a runtime trace record?
Answer
The scheduler, GC and goroutine events with timestamps
It answers latency questions.
-
Which tool reads a trace file?
Answer
go tool trace
It opens a timeline page.
-
What does importing net/http/pprof do?
Answer
Registers handlers on http.DefaultServeMux
The blank import is the switch.
-
Why mount pprof on a private port?
Answer
A heap profile and a goroutine dump are not for the internet
They reveal internals.
-
What does a climbing goroutine count suggest?
Answer
A leak
Something never ends.
-
Which path dumps every goroutine with its stack?
Answer
/debug/pprof/goroutine?debug=2
It is the first look at a stuck server.
The code on this page was run with go 1.26.10 on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev