Capstone 2: benchmarks and profiles
The store writes a whole JSON file on every change, so its cost is the cost of a file write, not of a map. The benchmark measures that cost and the profile shows where it goes. Both are needed: the number says how slow it is, the profile says what to do about it.
The benchmark
The first benchmark puts one task a thousand times into a file in a temporary directory, and the second reads a hundred-task file. The timer is reset after the setup, so the loop measures the operation and not the fixture. b.TempDir gives each benchmark its own directory.
package tasks
import (
"path/filepath"
"testing"
)
func BenchmarkFileStorePut(b *testing.B) {
store := NewFileStore(filepath.Join(b.TempDir(), "tasks.json"))
task := Task{ID: 1, Title: "buy milk"}
b.ResetTimer()
for i := 0; i < b.N; i++ {
if err := store.Put(task); err != nil {
b.Fatal(err)
}
}
}
func BenchmarkFileStoreAll(b *testing.B) {
store := NewFileStore(filepath.Join(b.TempDir(), "tasks.json"))
for i := 1; i <= 100; i++ {
if err := store.Put(Task{ID: i, Title: "task"}); err != nil {
b.Fatal(err)
}
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
if _, err := store.All(); err != nil {
b.Fatal(err)
}
}
}
A benchmark of a store that silently ignored errors would measure nothing, so every call is checked and a failure ends the benchmark. The two operations are the two costs of the command: a write per change and a read per list.
The numbers
The run below is from this container. A put costs about 224 microseconds and 32 allocations, and a list of a hundred tasks about 212 microseconds because it reads and parses the whole file before returning. The numbers move a few percent between runs, and the disk of the machine is part of the measurement.
go test -bench=. -benchmem ./tasks
goos: linux
goarch: amd64
pkg: example.com/tasks/tasks
cpu: Intel(R) Core(TM) i3-6006U CPU @ 2.00GHz
BenchmarkFileStorePut-4 5271 224330 ns/op 2392 B/op 32 allocs/op
BenchmarkFileStoreAll-4 5416 212643 ns/op 21232 B/op 121 allocs/op
PASS
ok example.com/tasks/tasks 2.514s
The time is dominated by the two file operations of the atomic save, the write and the rename, and by the read on every load. The allocations are the JSON document and the temporary path, which is a small part of the cost and the part a program can fix; the syscalls are the part it cannot.
The profile
-cpuprofile writes a profile of the benchmark, and go tool pprof -top prints the functions that used the CPU. The first line is the runtime's system call entry, which is what a file write looks like in a profile: the process is waiting in the kernel, not computing.
go test -run=^$ -bench=BenchmarkFileStorePut -cpuprofile=/tmp/tasks-cpu.out ./tasks
go tool pprof -top -nodecount=5 /tmp/tasks-cpu.out
File: tasks.test
Type: cpu
Showing nodes accounting for 1170ms, 88.64% of 1320ms total
flat flat% sum% cum cum%
1080ms 81.82% 81.82% 1080ms 81.82% internal/runtime/syscall/linux.Syscall6
30ms 2.27% 84.09% 30ms 2.27% reflect.(*rtype).NumMethod
20ms 1.52% 85.61% 40ms 3.03% encoding/json.checkValid
20ms 1.52% 87.12% 20ms 1.52% runtime.(*spanInlineMarkBits).init
The profile says 82 percent of the sampled CPU is in the system call wrapper, so the program is not computing, it is waiting for the disk. Optimising the JSON encoder would not change the number; batching the writes or keeping the tasks in memory would. That is the decision the profile makes obvious and the benchmark alone does not.
Why a file store is slow
Every command in this capstone runs once and exits, so 224 microseconds per write is invisible to a user: the process starts, the store reads, writes and exits. A server that wrote a file per request would not survive a hundred requests a second, which is why capstone 3 keeps its notes in memory or in one file behind a mutex and says where a database would replace it. The benchmark is the evidence for that sentence.
Measuring a store
Measure the operation the user performs, with the fixture the user has: a hundred tasks, not one. Reset the timer after the setup. Check every error, or the benchmark measures a no-op. Run the profile on the benchmark that is too slow, not on all of them, and read the top ten lines before changing the code. The next lesson builds and stamps the release.
Exercises
- Benchmark a put into an in-memory store with the same interface and compare. What is the ratio, and what does the next command in the capstone pay for durability?
- Run the benchmark with -count=5 and compare the five lines. What is the spread, and is the disk cache warm after the first?
- Profile BenchmarkFileStoreAll and read the top lines. Which function dominates, and what would reduce it?
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
What does a put cost in the run?
Answer
About 224 microseconds and 32 allocations
The two file operations dominate.
-
Why reset the timer after the setup?
Answer
The loop measures the operation, not the fixture
b.ResetTimer is for that.
-
What does the profile show at the top?
Answer
The system call wrapper, because the process waits for the disk
A file write is kernel time.
-
What would optimising the encoder change?
Answer
Almost nothing, the time is in the syscalls
The profile says where the time is.
-
Why is the store acceptable for a CLI?
Answer
The command runs once and exits
A server would need another store.
-
Why measure a hundred tasks and not one?
Answer
The fixture should match the user's data
The list reads and parses all of them.
-
What does -cpuprofile write?
Answer
A profile of the benchmark run
go tool pprof reads it.
-
What does a benchmark with ignored errors measure?
Answer
A no-op
Check every call.
-
Which change would make the store faster?
Answer
Keeping the tasks in memory and writing less often
The syscalls are the cost.
-
What does the next lesson do?
Answer
Build and stamp the release
The last lesson of capstone 2.
The code on this page was run with go 1.26.10 on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev