Capstone 2: benchmarks and profiles

The store writes a whole JSON file on every change, so its cost is the cost of a file write, not of a map. The benchmark measures that cost and the profile shows where it goes. Both are needed: the number says how slow it is, the profile says what to do about it.

The benchmark

The first benchmark puts one task a thousand times into a file in a temporary directory, and the second reads a hundred-task file. The timer is reset after the setup, so the loop measures the operation and not the fixture. b.TempDir gives each benchmark its own directory.

package tasks import ( "path/filepath" "testing" ) func BenchmarkFileStorePut(b *testing.B) { store := NewFileStore(filepath.Join(b.TempDir(), "tasks.json")) task := Task{ID: 1, Title: "buy milk"} b.ResetTimer() for i := 0; i < b.N; i++ { if err := store.Put(task); err != nil { b.Fatal(err) } } } func BenchmarkFileStoreAll(b *testing.B) { store := NewFileStore(filepath.Join(b.TempDir(), "tasks.json")) for i := 1; i <= 100; i++ { if err := store.Put(Task{ID: i, Title: "task"}); err != nil { b.Fatal(err) } } b.ResetTimer() for i := 0; i < b.N; i++ { if _, err := store.All(); err != nil { b.Fatal(err) } } }

A benchmark of a store that silently ignored errors would measure nothing, so every call is checked and a failure ends the benchmark. The two operations are the two costs of the command: a write per change and a read per list.

The numbers

The run below is from this container. A put costs about 224 microseconds and 32 allocations, and a list of a hundred tasks about 212 microseconds because it reads and parses the whole file before returning. The numbers move a few percent between runs, and the disk of the machine is part of the measurement.

go test -bench=. -benchmem ./tasks

goos: linux
goarch: amd64
pkg: example.com/tasks/tasks
cpu: Intel(R) Core(TM) i3-6006U CPU @ 2.00GHz
BenchmarkFileStorePut-4 5271 224330 ns/op 2392 B/op 32 allocs/op
BenchmarkFileStoreAll-4 5416 212643 ns/op 21232 B/op 121 allocs/op
PASS
ok example.com/tasks/tasks 2.514s

The time is dominated by the two file operations of the atomic save, the write and the rename, and by the read on every load. The allocations are the JSON document and the temporary path, which is a small part of the cost and the part a program can fix; the syscalls are the part it cannot.

The profile

-cpuprofile writes a profile of the benchmark, and go tool pprof -top prints the functions that used the CPU. The first line is the runtime's system call entry, which is what a file write looks like in a profile: the process is waiting in the kernel, not computing.

go test -run=^$ -bench=BenchmarkFileStorePut -cpuprofile=/tmp/tasks-cpu.out ./tasks

go tool pprof -top -nodecount=5 /tmp/tasks-cpu.out

File: tasks.test
Type: cpu
Showing nodes accounting for 1170ms, 88.64% of 1320ms total
flat flat% sum% cum cum%
1080ms 81.82% 81.82% 1080ms 81.82% internal/runtime/syscall/linux.Syscall6
30ms 2.27% 84.09% 30ms 2.27% reflect.(*rtype).NumMethod
20ms 1.52% 85.61% 40ms 3.03% encoding/json.checkValid
20ms 1.52% 87.12% 20ms 1.52% runtime.(*spanInlineMarkBits).init

The profile says 82 percent of the sampled CPU is in the system call wrapper, so the program is not computing, it is waiting for the disk. Optimising the JSON encoder would not change the number; batching the writes or keeping the tasks in memory would. That is the decision the profile makes obvious and the benchmark alone does not.

Why a file store is slow

Every command in this capstone runs once and exits, so 224 microseconds per write is invisible to a user: the process starts, the store reads, writes and exits. A server that wrote a file per request would not survive a hundred requests a second, which is why capstone 3 keeps its notes in memory or in one file behind a mutex and says where a database would replace it. The benchmark is the evidence for that sentence.

Measuring a store

Measure the operation the user performs, with the fixture the user has: a hundred tasks, not one. Reset the timer after the setup. Check every error, or the benchmark measures a no-op. Run the profile on the benchmark that is too slow, not on all of them, and read the top ten lines before changing the code. The next lesson builds and stamps the release.

Exercises

  1. Benchmark a put into an in-memory store with the same interface and compare. What is the ratio, and what does the next command in the capstone pay for durability?
  2. Run the benchmark with -count=5 and compare the five lines. What is the spread, and is the disk cache warm after the first?
  3. Profile BenchmarkFileStoreAll and read the top lines. Which function dominates, and what would reduce it?

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. What does a put cost in the run?

    Answer

    About 224 microseconds and 32 allocations

    The two file operations dominate.

  2. Why reset the timer after the setup?

    Answer

    The loop measures the operation, not the fixture

    b.ResetTimer is for that.

  3. What does the profile show at the top?

    Answer

    The system call wrapper, because the process waits for the disk

    A file write is kernel time.

  4. What would optimising the encoder change?

    Answer

    Almost nothing, the time is in the syscalls

    The profile says where the time is.

  5. Why is the store acceptable for a CLI?

    Answer

    The command runs once and exits

    A server would need another store.

  6. Why measure a hundred tasks and not one?

    Answer

    The fixture should match the user's data

    The list reads and parses all of them.

  7. What does -cpuprofile write?

    Answer

    A profile of the benchmark run

    go tool pprof reads it.

  8. What does a benchmark with ignored errors measure?

    Answer

    A no-op

    Check every call.

  9. Which change would make the store faster?

    Answer

    Keeping the tasks in memory and writing less often

    The syscalls are the cost.

  10. What does the next lesson do?

    Answer

    Build and stamp the release

    The last lesson of capstone 2.

The code on this page was run with go 1.26.10 on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev

Banner image