Capstone 1: table tests and fuzzing
The library has three test functions and a fuzz target. The table tests check the tokens and the documents the author thought of; the fuzz target checks the property the two halves must satisfy for every input, which is that whatever parses can be written and parsed again. The files live next to the package, as Go tests do.
The lexer table
The first test feeds three short documents through the lexer and compares the sequence of kinds, and for two of them the exact texts. A failure names the document, the index and both kinds, which is the whole reason for the table shape: one case per line and a message that points at the case.
package jsonx
import (
"io"
"testing"
)
func TestLexer(t *testing.T) {
tests := []struct {
src string
kinds []Kind
texts []string
}{
{`{}`, []Kind{BraceOpen, BraceClose}, []string{"{", "}"}},
{`[1, -2.5e3]`, []Kind{BracketOpen, Number, Comma, Number, BracketClose}, []string{"[", "1", ",", "-2.5e3", "]"}},
{`{"a": true, "b": false, "c": null}`, []Kind{BraceOpen, String, Colon, True, Comma, String, Colon, False, Comma, String, Colon, Null, BraceClose}, nil},
}
for _, tt := range tests {
lex := NewLexer([]byte(tt.src))
var kinds []Kind
var texts []string
for {
tok, err := lex.Next()
if err == io.EOF {
break
}
if err != nil {
t.Fatalf("%s: %v", tt.src, err)
}
kinds = append(kinds, tok.Kind)
texts = append(texts, tok.Text)
}
if len(kinds) != len(tt.kinds) {
t.Fatalf("%s: got %d tokens, want %d", tt.src, len(kinds), len(tt.kinds))
}
for i := range kinds {
if kinds[i] != tt.kinds[i] {
t.Errorf("%s token %d: got %s, want %s", tt.src, i, kinds[i], tt.kinds[i])
}
if tt.texts != nil && texts[i] != tt.texts[i] {
t.Errorf("%s token %d: got %q, want %q", tt.src, i, texts[i], tt.texts[i])
}
}
}
}
The expected texts are nil for the third case because that case only cares about the kinds. The test fails fast on a count mismatch and then reports every kind that differs, so a run shows all the problems in the table and not only the first.
Valid and invalid documents
The second file has a table of documents and the encoding each must produce, and a table of broken documents with a substring each error must contain. The valid table checks the writer at the same time, because the expected value is the compact encoding; the invalid table checks both the message and that it starts with a position.
package jsonx
import (
"strings"
"testing"
)
func TestParseValid(t *testing.T) {
tests := []struct {
src string
want string
}{
{`null`, `null`},
{`true`, `true`},
{`42`, `42`},
{`"hi"`, `"hi"`},
{`[1,2,3]`, `[1,2,3]`},
{`{"b":2,"a":1}`, `{"a":1,"b":2}`},
{`{"s":"a\nb"}`, `{"s":"a\nb"}`},
}
for _, tt := range tests {
v, err := Parse([]byte(tt.src))
if err != nil {
t.Errorf("%s: %v", tt.src, err)
continue
}
if got := string(v.JSON()); got != tt.want {
t.Errorf("%s: got %s, want %s", tt.src, got, tt.want)
}
}
}
func TestParseInvalid(t *testing.T) {
tests := []struct {
src string
msg string
}{
{`{"a": }`, "unexpected }"},
{`{"a": 1,}`, "trailing comma"},
{`[1, 2`, "unterminated array"},
{`nul`, "invalid literal"},
{`"open`, "unterminated string"},
{`{"a" 1}`, "expected ':'"},
}
for _, tt := range tests {
_, err := Parse([]byte(tt.src))
if err == nil {
t.Errorf("%s: expected an error", tt.src)
continue
}
if !strings.Contains(err.Error(), tt.msg) {
t.Errorf("%s: error %q does not contain %q", tt.src, err, tt.msg)
}
if !strings.HasPrefix(err.Error(), "line ") {
t.Errorf("%s: error has no position: %q", tt.src, err)
}
}
}
The invalid table does not compare the whole message, because the wording may improve; it compares the part that carries the meaning and asserts the position prefix separately. The valid table compares the whole encoding, because that output is the contract.
The fuzz target
A fuzz target takes the type it fuzzes and adds a seed corpus. The body below ignores a parse error, since most random bytes are not JSON, and for a document that parses it writes the value and parses the bytes again. The second parse must succeed and produce the same encoding, which is the round-trip property the writer and the parser promise each other.
package jsonx
import "testing"
func FuzzParse(f *testing.F) {
for _, seed := range []string{`null`, `[1,2]`, `{"a":true}`, `"x"`} {
f.Add([]byte(seed))
}
f.Fuzz(func(t *testing.T, data []byte) {
v, err := Parse(data)
if err != nil {
return
}
round, err := Parse(v.JSON())
if err != nil {
t.Fatalf("round trip failed: %v", err)
}
if string(round.JSON()) != string(v.JSON()) {
t.Fatalf("round trip changed %s to %s", v.JSON(), round.JSON())
}
})
}
The four seeds are the four kinds a document can start with: a literal, an array, an object and a string. A fuzzer mutates them, and when it finds a panic or a failed round trip it writes the input to testdata/fuzz and reports it, which turns a discovered bug into a regression test with no work from the author.
Running them
go test -v ./jsonx runs the three tests and the fuzz seeds and prints each name. The seed cases of the fuzz target run as ordinary subtests, which is why the output lists FuzzParse/seed#0 through seed#3.
go test -v ./jsonx
=== RUN TestParseValid
--- PASS: TestParseValid (0.00s)
=== RUN TestParseInvalid
--- PASS: TestParseInvalid (0.00s)
=== RUN TestLexer
--- PASS: TestLexer (0.00s)
=== RUN FuzzParse
=== RUN FuzzParse/seed#0
=== RUN FuzzParse/seed#1
=== RUN FuzzParse/seed#2
=== RUN FuzzParse/seed#3
--- PASS: FuzzParse (0.00s)
--- PASS: FuzzParse/seed#0 (0.00s)
--- PASS: FuzzParse/seed#1 (0.00s)
--- PASS: FuzzParse/seed#2 (0.00s)
--- PASS: FuzzParse/seed#3 (0.00s)
PASS
ok example.com/jsonx/jsonx 0.004s
-fuzz turns the target into a running fuzzer and needs a time limit. Two seconds produced about forty thousand executions on four workers; the count depends on the machine and on how fast the inputs fail, so the number below is this run and not a benchmark.
go test -run=^$ -fuzz=FuzzParse -fuzztime=2s ./jsonx
fuzz: elapsed: 0s, gathering baseline coverage: 84/84 completed, now fuzzing with 4 workers
fuzz: elapsed: 2s, execs: 39820 (18931/sec), new interesting: 17 (total: 101)
PASS
ok example.com/jsonx/jsonx 2.132s
The baseline count of 84 inputs comes from the seeds plus the entries an earlier run left in testdata/fuzz; a fresh checkout starts with the four seeds. The fuzzer found no failure in this run, which is not a proof: it is evidence that the property held for forty thousand inputs.
What the tests promise
A table test promises the cases it lists. A fuzz target promises a property for every input the fuzzer reaches. Together they cover the two failure modes: a regression in a known case and a crash in an unknown one. The corpus in testdata/fuzz is committed with the fix, so the second failure mode becomes the first.
The next lesson wraps the library in a command with flags, a file argument and exit codes, which is where the errors of the last lesson reach a shell.
Exercises
- Add a valid case with a unicode escape and one with a nested empty object. Which table do they belong to, and what does the expected encoding look like?
- Run the fuzzer for ten seconds and read the summary. How many executions ran, and what does the ratio say about the speed of the parser?
- Break the writer on purpose, for example by not escaping a quote, and run the fuzzer. How quickly does it find an input, and what does it write to testdata?
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
Where do the test files live?
Answer
Next to the package, with a _test.go suffix
Go tests compile with the package.
-
What does a table test make easy?
Answer
One case per line and a message that names the case
The failure points at the row.
-
Why compare a substring of an error?
Answer
The wording may improve, the meaning may not
The test asserts the meaningful part.
-
What is a fuzz seed?
Answer
An input the fuzzer starts from
The corpus grows from the seeds.
-
What property does FuzzParse check?
Answer
A document that parses can be written and parsed again unchanged
It is the round trip.
-
Where does a failing fuzz input go?
Answer
testdata/fuzz, as a regression case
The bug becomes a test.
-
Why do the seeds appear in go test -v?
Answer
They run as ordinary subtests
FuzzParse/seed#0 and so on.
-
Does a clean fuzz run prove correctness?
Answer
No, it is evidence for the inputs it tried
A property is not a proof.
-
What does -fuzztime set?
Answer
How long the fuzzer runs
A fuzzer needs a limit.
-
Why commit the fuzz corpus?
Answer
The discovered inputs become regression tests
A found bug stays found.
The code on this page was run with go 1.26.10 on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev