Runtime metrics and traces

A Go process already knows most of what a dashboard wants: how many goroutines run, how much the heap holds, how often the collector ran, how long a pause was. runtime/metrics exposes them by name with a stable path, runtime/trace records what the scheduler did, and net/http/pprof serves the profiles over HTTP. This lesson reads each one from a real run.

runtime/metrics

A metric has a name such as /sched/goroutines:goroutines and a value whose kind is part of the name after the colon. The program describes the samples it wants, calls Read once and prints them. The numbers below are this run; the names are the stable part.

package main import ( "fmt" "runtime/metrics" ) func main() { samples := []metrics.Sample{ {Name: "/sched/goroutines:goroutines"}, {Name: "/gc/heap/allocs:bytes"}, {Name: "/gc/cycles/total:gc-cycles"}, } metrics.Read(samples) for _, s := range samples { fmt.Printf("%s = %v\n", s.Name, s.Value.Uint64()) } }

go build -o app main.go && ./app

/sched/goroutines:goroutines = 6
/gc/heap/allocs:bytes = 57216
/gc/cycles/total:gc-cycles = 0

Six goroutines because the metrics package starts a few of its own, and no collection yet because the program is small. A service records these names on an interval and exports them to Prometheus with the same names, which is why the paths are versioned and documented. metrics.All lists every name a build supports, which is how a program discovers one it did not know.

MemStats

The older API is a struct with every counter. It is still the one most dashboards use and the one a program reads when it wants the heap and the collector in one call. ReadMemStats stops the world briefly, so it belongs on a timer and not in a handler.

package main import ( "fmt" "runtime" ) func main() { var stats runtime.MemStats runtime.ReadMemStats(&stats) fmt.Println("cpus:", runtime.NumCPU()) fmt.Println("goroutines:", runtime.NumGoroutine()) fmt.Println("heap alloc > 0:", stats.Alloc > 0) fmt.Println("total alloc > 0:", stats.TotalAlloc > 0) }

go build -o app main.go && ./app

cpus: 4
goroutines: 1
heap alloc > 0: true
total alloc > 0: true

A program at rest has one goroutine and a small heap. The booleans are there because the exact numbers change with every build; the counters that matter in production are Alloc, HeapInuse, NumGC and the pause percentiles, and a service exports them with its own labels.

A runtime trace

A trace records what the scheduler, the garbage collector and the goroutines did, with timestamps. The program starts a trace into a temporary file, runs three goroutines that sum a thousand numbers, stops the trace and checks that the file has bytes in it. The file is read with go tool trace.

package main import ( "fmt" "os" "runtime/trace" "sync" ) func main() { f, err := os.CreateTemp("", "trace-*.out") if err != nil { panic(err) } defer os.Remove(f.Name()) defer f.Close() if err := trace.Start(f); err != nil { panic(err) } var wg sync.WaitGroup for i := 0; i < 3; i++ { wg.Add(1) go func() { defer wg.Done() sum := 0 for j := 0; j < 1000; j++ { sum += j } _ = sum }() } wg.Wait() trace.Stop() info, err := f.Stat() if err != nil { panic(err) } fmt.Println("trace bytes > 0:", info.Size() > 0) }

go build -o app main.go && ./app

trace bytes > 0: true

The file is small and real, and go tool trace trace.out opens a page that shows the goroutines over time, the GC cycles and the scheduler latency. A trace is the tool for a latency question that a profile cannot answer: not where the CPU went, but what was runnable and did not run.

The pprof endpoints

Importing net/http/pprof registers handlers on http.DefaultServeMux: the index, the goroutine dump, the heap profile, the CPU profile and the trace. The program serves that mux and asks for the index, which lists the profiles and mentions goroutine among them.

package main import ( "fmt" "io" "net/http" "net/http/httptest" "strings" _ "net/http/pprof" ) func main() { srv := httptest.NewServer(http.DefaultServeMux) defer srv.Close() resp, err := http.Get(srv.URL + "/debug/pprof/") if err != nil { panic(err) } body, _ := io.ReadAll(resp.Body) resp.Body.Close() fmt.Println("status:", resp.Status) fmt.Println("mentions goroutine:", strings.Contains(string(body), "goroutine")) }

go build -o app main.go && ./app

status: 200 OK
mentions goroutine: true

The blank import is the switch, and a service mounts the handlers on a private port rather than on the public mux, because a heap profile and a goroutine dump are not for the internet. The path /debug/pprof/goroutine?debug=2 prints every goroutine and the line it waits on, which is the first thing to read when a server stops answering.

What to watch

Five numbers cover a Go service: goroutines over time, heap in use, GC cycles per second, the pause percentiles and the number of open connections. A goroutine count that climbs is a leak. A heap that climbs without a limit is a leak or a cache without one. A pause in the milliseconds is normal, and a pause in the hundreds is a signal to look. Everything else is a detail of the workload.

The lesson after the next one ends Part 9; the capstone project then uses these tools on a parser, with a benchmark and a profile of its own.

Exercises

  1. Add /sched/latencies:seconds to the samples and print its kind. Which accessor reads it, and why is it not a Uint64?
  2. Run the trace program with ten goroutines and open the file with go tool trace. What does the timeline show that the printed line does not?
  3. Serve the pprof mux on a listener and fetch /debug/pprof/goroutine?debug=2. How many goroutines does the dump list, and which one is the HTTP handler?

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. What is the name of a runtime metric?

    Answer

    A path with the kind after a colon

    The path is stable across versions.

  2. How is a metric value read?

    Answer

    By the accessor of its kind, such as Uint64

    The kind is part of the name.

  3. Why is ReadMemStats not for a handler?

    Answer

    It stops the world briefly

    A timer is the place for it.

  4. Which counters matter most in a service?

    Answer

    Goroutines, heap, GC cycles and pause percentiles

    They show leaks and pressure.

  5. What does a runtime trace record?

    Answer

    The scheduler, GC and goroutine events with timestamps

    It answers latency questions.

  6. Which tool reads a trace file?

    Answer

    go tool trace

    It opens a timeline page.

  7. What does importing net/http/pprof do?

    Answer

    Registers handlers on http.DefaultServeMux

    The blank import is the switch.

  8. Why mount pprof on a private port?

    Answer

    A heap profile and a goroutine dump are not for the internet

    They reveal internals.

  9. What does a climbing goroutine count suggest?

    Answer

    A leak

    Something never ends.

  10. Which path dumps every goroutine with its stack?

    Answer

    /debug/pprof/goroutine?debug=2

    It is the first look at a stuck server.

The code on this page was run with go 1.26.10 on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev