Capstone 1: benchmarks against encoding/json
A benchmark is a function named BenchmarkXxx(b *testing.B) in a test file. The runner calls it with a growing b.N until the measurement is long enough, and reports nanoseconds per operation, bytes per operation and allocations per operation. The two benchmarks below parse the same small document with this library and with encoding/json.
The benchmark
The body of a benchmark is the operation the number describes, in a loop of b.N. The error check is outside the interesting path but inside the loop, so the compiler cannot remove the call. b.Fatal stops the benchmark on a failure, which is the only way a wrong result can make a benchmark invalid.
package jsonx
import (
"encoding/json"
"testing"
)
var sample = []byte(`{"id":1,"title":"buy milk","done":false,"tags":["home","errand"],"items":[1,2,3,4,5]}`)
func BenchmarkParseSmall(b *testing.B) {
for i := 0; i < b.N; i++ {
if _, err := Parse(sample); err != nil {
b.Fatal(err)
}
}
}
func BenchmarkStdlibSmall(b *testing.B) {
for i := 0; i < b.N; i++ {
var v any
if err := json.Unmarshal(sample, &v); err != nil {
b.Fatal(err)
}
}
The stdlib benchmark decodes into an any, which is the comparable job: both parsers have to read the document and build a representation. A benchmark against a struct decode would measure a different thing, because the stdlib would skip the map it builds here.
The numbers
-benchmem adds the bytes and allocations columns, which matter more than the time for a parser: an allocation is a garbage-collection cost that the ns/op hides. The run below is from this container, on four CPUs, and the numbers move by a few percent between runs.
go test -bench=. -benchmem ./jsonx
goos: linux
goarch: amd64
pkg: example.com/jsonx/jsonx
cpu: Intel(R) Core(TM) i3-6006U CPU @ 2.00GHz
BenchmarkParseSmall-4 116889 8813 ns/op 3552 B/op 36 allocs/op
BenchmarkStdlibSmall-4 280617 4052 ns/op 1008 B/op 31 allocs/op
PASS
ok example.com/jsonx/jsonx 2.329s
This parser takes about 8.8 microseconds per document against 4.1 for the standard library, uses 3.5 times the bytes and five more allocations. The ratio is roughly 2.2 times slower on this input, which is the honest result of the comparison and not a defect to hide.
Why the standard library wins
Four differences explain the gap. encoding/json caches a per-type plan, so it does not walk the type on every call; this parser walks a map and sorts the keys on every object. The stdlib decodes into any with a small set of concrete types; this one copies every string into a struct field of a fixed 64-byte value. The stdlib has been tuned for a decade, including its own scanner that avoids the token objects. And this library keeps positions and a kind per value, which the benchmark pays for even though it does not print them.
A fair comparison would also measure the features: the stdlib does not give a position for an error, a sorted object encoding from a map (it sorts map keys too), or a value type with total accessors. The benchmark says "this library is slower"; it does not say "this library is worse", and a program that parses a config once at start-up should use whichever API it prefers.
What the benchmark does not say
A microbenchmark parses the same small document in a loop, so it measures the hot path with a warm cache and no network. A service parses a body per request, holds it for a millisecond and allocates around it; the parser is a small part of the profile. And a benchmark on a laptop is not a benchmark on the server, which is why a real comparison runs in the deployment image with the production input.
The useful part of the number is the allocations: 36 per document is where a profile would look first, and the escape analysis of lesson 109 explains which of them are avoidable. The next part starts a second capstone, a tasks CLI with storage, and its benchmark lesson applies the same method to a different workload.
Measuring honestly
Run the benchmark on the machine that matters, with -benchmem and -count=5, and compare with a tool rather than by eye. Report the input, the version and the CPU with the number. And measure the feature you are proud of as well as the one you are not: a parser that is twice as slow but reports the line and column of an error is a trade a program can make on purpose.
Exercises
- Add a benchmark for a document ten times larger and run both. Does the ratio change, and what does that say about the fixed cost of a call?
- Run with -count=5 and compare the five lines. What is the spread, and why is one run not a measurement?
- Remove the sort from ObjectValue and benchmark again. How much time does the sort cost, and what does the output lose?
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
What is a benchmark function named?
Answer
BenchmarkXxx(b *testing.B)
The runner finds it by the prefix.
-
What is b.N?
Answer
The number of iterations the runner chose
It grows until the measurement is long enough.
-
What does -benchmem add?
Answer
Bytes and allocations per operation
Allocations are the GC cost.
-
How much slower was this parser here?
Answer
About 2.2 times
8813 against 4052 ns/op.
-
Why does the standard library win?
Answer
A cached type plan and a tuned scanner among other reasons
The page lists four differences.
-
What does the benchmark not measure?
Answer
A request lifecycle around the parse
It is a microbenchmark.
-
Which column matters most for a parser?
Answer
Allocations per operation
They drive the GC cost.
-
Why run a benchmark several times?
Answer
The spread between runs is the noise
One number is not a measurement.
-
Where should a real comparison run?
Answer
On the deployment image with production input
The machine and input change the result.
-
What is a slower parser with positions good for?
Answer
Errors a user can act on
It is a trade made on purpose.
The code on this page was run with go 1.26.10 on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev