Benchmarks and benchstat

A benchmark measures a function by running it a number of times the runner chooses. The benchmark function receives a *testing.B and loops b.N times; the runner raises N until the measurement lasts long enough to be meaningful. testing.Benchmark runs the same loop from a normal program, which is the quick way to compare two implementations without a test file. The fuzzing lesson used the same testing package for a different kind of generated input.

A measurement inside a program

The two functions build the same string: the first concatenates in a loop, the second appends to a strings.Builder. testing.Benchmark returns a BenchmarkResult whose String prints the operations and the time per operation, and AllocsPerOp reports the allocations. The comparison is the lesson: the builder avoids the copy of the growing string.

package main import ( "fmt" "strings" "testing" ) func Join(parts []string) string { out := "" for _, p := range parts { out += p + "," } return out } func Build(parts []string) string { var b strings.Builder for _, p := range parts { b.WriteString(p) b.WriteByte(',') } return b.String() } func main() { parts := make([]string, 1000) for i := range parts { parts[i] = "item" } join := testing.Benchmark(func(b *testing.B) { for i := 0; i < b.N; i++ { Join(parts) } }) build := testing.Benchmark(func(b *testing.B) { for i := 0; i < b.N; i++ { Build(parts) } }) fmt.Println("Join: ", join) fmt.Println("Build:", build) fmt.Println("Join allocations: ", join.AllocsPerOp()) fmt.Println("Build allocations:", build.AllocsPerOp()) }

go run bench.go

Join: 788 1563531 ns/op
Build: 56486 23423 ns/op
Join allocations: 1000
Build allocations: 13

The numbers are one run on this machine, so treat the ratio as the finding and the digits as noise: the builder is faster and allocates far less, and that ordering is what a benchmark is for. A single measurement is not proof of a regression or a fix; the same code on another CPU, another Go version or another load gives other numbers, which is what the last section is about.

A benchmark in a test file

A test file holds benchmarks next to the tests, and go test -bench=. -run=^$ runs them without running the tests: the -run pattern matches nothing. b.ResetTimer discards the setup from the measurement, and b.ReportAllocs adds the B/op and allocs/op columns that -benchmem also turns on.

package text import ( "strings" "testing" ) func Join(parts []string) string { out := "" for _, p := range parts { out += p + "," } return out } func Build(parts []string) string { var b strings.Builder for _, p := range parts { b.WriteString(p) b.WriteByte(',') } return b.String() } func benchParts() []string { parts := make([]string, 1000) for i := range parts { parts[i] = "item" } return parts } func BenchmarkJoin(b *testing.B) { parts := benchParts() b.ResetTimer() b.ReportAllocs() for i := 0; i < b.N; i++ { Join(parts) } } func BenchmarkBuild(b *testing.B) { parts := benchParts() b.ResetTimer() b.ReportAllocs() for i := 0; i < b.N; i++ { Build(parts) } }

go test -bench=. -benchmem -run=^$ .

goos: linux
goarch: amd64
pkg: example.com/text
cpu: Intel(R) Core(TM) i3-6006U CPU @ 2.00GHz
BenchmarkJoin-4 925 1549096 ns/op 2684245 B/op 1000 allocs/op
BenchmarkBuild-4 54986 32059 ns/op 17912 B/op 13 allocs/op
PASS
ok example.com/text 3.575s

The -4 after the package name is the default GOMAXPROCS, and the two lines after each name are the time and the allocation columns. A benchmark that allocates in the loop reports bytes per operation; a benchmark that does not says 0 B/op and 0 allocs/op, which is the cheapest kind of result to defend in a review.

Comparing two runs

benchstat compares two files of benchmark output and reports the change with a confidence range, which is the answer to "is this 5% difference real?". The workflow is to run the benchmark several times, save the output with -count=10, then compare the old and the new file. It is a separate tool, not part of the toolchain, and the numbers it needs are exactly the unstable ones a single run prints.

A benchmark is a measurement of a program, not a proof of correctness: it does not check the result, so the function still needs its test. Keep the benchmark next to the test, keep the setup out of the timed loop, and compare two versions rather than reading one number as a verdict.

Exercises

  1. Add a benchmark that calls Join with ten parts instead of a thousand. Does the builder still win by the same factor?
  2. Run the benchmark with -count=5 and look at the spread of the ns/op column. Would you trust a 3% difference from one run?
  3. Write a benchmark for a function that allocates a slice per call, with and without a size hint. Which column shows the difference most clearly?

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. What does a benchmark loop over?

    Answer

    b.N, the number of iterations the runner chooses

    The runner raises N until the measurement is long enough.

  2. What does b.ResetTimer do?

    Answer

    Discards the work done before the loop from the measurement

    Setup belongs outside the timed section, and ResetTimer marks the boundary.

  3. What does -benchmem add?

    Answer

    The bytes and allocations per operation

    b.ReportAllocs does the same for one benchmark.

  4. What does testing.Benchmark let a program do?

    Answer

    Run a benchmark loop and read the result as a value

    It returns a BenchmarkResult with String and AllocsPerOp.

  5. Why does -run=^$ appear with -bench?

    Answer

    It matches no test, so only the benchmarks run

    Without it the tests run too, which costs time and adds noise.

  6. What is a single benchmark number good for?

    Answer

    A rough comparison, not a verdict on a small change

    The measurement varies with machine, load and compiler version.

  7. What does benchstat do?

    Answer

    Compares two benchmark result files with a confidence range

    It is the tool for deciding whether a difference is real.

  8. Which result looks best in -benchmem?

    Answer

    0 B/op and 0 allocs/op for the same work

    Fewer allocations per operation is the usual goal.

  9. What is wrong with benchmarking a function that is also wrong?

    Answer

    The benchmark measures the wrong work and proves nothing about correctness

    A benchmark has no assertion; the test is still required.

  10. What does the -4 in the package line of a benchmark run mean?

    Answer

    The value of GOMAXPROCS used for the run

    It comes from the CPU count unless GOMAXPROCS overrides it.

The code on this page was run with go 1.26.10 on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev