Streaming JSON with Decoder and Encoder
A document is one JSON value and a stream is many of them one after another, which is how logs, exports and APIs that page through a million records are sent. json.Decoder reads one value per Decode call and stops exactly after it, so the next call starts at the next value; json.Encoder writes one value and a newline, which is the other half of the same format.
Decoder in a loop
The input below is two objects, one per line, with no array around them. Each Decode reads one and the loop ends at io.EOF, which is the normal end and not a failure. A decoder reads from any io.Reader, so the same loop runs over a file, a response body or a socket.
package main
import (
"encoding/json"
"fmt"
"io"
"strings"
)
func main() {
stream := `{"id":1,"name":"a"}
{"id":2,"name":"b"}
`
dec := json.NewDecoder(strings.NewReader(stream))
for {
var m map[string]any
err := dec.Decode(&m)
if err == io.EOF {
break
}
if err != nil {
fmt.Println("error:", err)
return
}
fmt.Println(m["id"], m["name"])
}
}
go build -o app main.go && ./app
1 a
2 b
The decoder ignored the newline between the objects because white space between values is legal. A file of one object per line is sometimes called JSON Lines, and a decoder needs no special mode for it; the strict version of the format forbids the newlines and is read with a single Unmarshal instead.
Encoder and its newline
Encoder.Encode marshals a value and appends a newline, which is why a stream written this way is readable and why a test that compares the bytes has to expect the newline. SetEscapeHTML(false) keeps < and & as they are, which matters for an API that sends code or HTML fragments.
package main
import (
"bytes"
"encoding/json"
"fmt"
)
func main() {
var buf bytes.Buffer
enc := json.NewEncoder(&buf)
enc.SetEscapeHTML(false)
if err := enc.Encode(map[string]any{"text": "a < b"}); err != nil {
panic(err)
}
if err := enc.Encode([]int{1, 2}); err != nil {
panic(err)
}
fmt.Print(buf.String())
}
go build -o app main.go && ./app
{"text":"a < b"}
[1,2]
Two values, two lines, and the less-than sign stayed a less-than sign. Without the call it would have been \u003c, which is valid JSON and decodes to the same string; a client that compares the raw bytes would see a difference that is not a difference in the data.
DisallowUnknownFields
A decoder can refuse a key the struct does not have, which is what an internal API between two services wants: a typo in a field name is a bug that should fail in the first test, not be silently dropped. The error names the field, and the value that was read before it stays in the struct.
package main
import (
"encoding/json"
"fmt"
"strings"
)
type User struct {
Name string `json:"name"`
}
func main() {
dec := json.NewDecoder(strings.NewReader(`{"name":"ada","age":36}`))
dec.DisallowUnknownFields()
var u User
err := dec.Decode(&u)
fmt.Println("user:", u)
fmt.Println("error:", err)
}
go build -o app main.go && ./app
user: {ada}
error: json: unknown field "age"
The name was filled and the decode failed on the next key. A server maps this error to a 400 and a client to a version mismatch. The default, which ignores the key, is the right choice for a public API where the server adds fields; the strict mode is for the case where the two sides are deployed together.
Token by token
Token reads the punctuation and the scalars of a document one at a time, with arrays and objects as json.Delim and numbers as float64 unless the decoder is in UseNumber mode. It is the tool for a document whose shape is not known in advance, such as a config with arbitrary keys, and it is rarely the tool for a struct that already describes the shape.
package main
import (
"encoding/json"
"fmt"
"io"
"strings"
)
func main() {
dec := json.NewDecoder(strings.NewReader(`[1, 2, {"a": true}]`))
for {
tok, err := dec.Token()
if err == io.EOF {
break
}
if err != nil {
fmt.Println("error:", err)
return
}
fmt.Printf("%T %v\n", tok, tok)
}
}
go build -o app main.go && ./app
json.Delim [
float64 1
float64 2
json.Delim {
string a
bool true
json.Delim }
json.Delim ]
The types are the point: the brackets are Delim, the numbers are floats and the key is a string. A program that walks tokens keeps its own stack to know whether a string is a key or a value, which is the work a struct would have done for it. The next lesson reads CSV, where the standard library also gives a reader and a writer.
Streams, not documents
A decoder is for a stream and an Unmarshal is for a document. The decoder keeps its position and its buffer, so a loop over a large file uses memory of one value; Unmarshal needs the whole input in memory and then the whole value. A response body from http.Response.Body is a stream, and a handler that decodes it with NewDecoder(r.Body).Decode(&req) never holds the body twice.
Two details save a debugging session. Decode returns io.EOF only when the input ended exactly between values, so a truncated document is an error and not an empty result. And a decoder reads ahead into its buffer, so a stream that mixes JSON with another protocol needs the decoder to own the reader for the whole of it.
Exercises
- Append a third object without a newline and run the loop. Does it decode, and what does the last call return?
- Encode the same map with and without SetEscapeHTML(false) and compare the bytes. Which key changed, and how many characters longer is the escaped one?
- Decode the strict example into a struct and print it after the error. Which fields were filled before the failure, and what does that say about a failed Unmarshal?
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
What does Decoder.Decode read?
Answer
One JSON value and stops after it
The next call starts at the next value.
-
What ends a Decode loop?
Answer
io.EOF between values
EOF is the normal end of a stream.
-
What does Encoder.Encode add after a value?
Answer
A newline
That is why a stream is one value per line.
-
What does SetEscapeHTML(false) keep?
Answer
The characters < > and & as they are
The default escapes them for HTML safety.
-
What does DisallowUnknownFields do?
Answer
Makes an unknown key an error
It is the strict mode of the decoder.
-
How are brackets returned by Token?
Answer
As json.Delim
That is how a walker tracks nesting.
-
What type is a number from Token?
Answer
float64, or json.Number in UseNumber mode
The same rule as Unmarshal into any.
-
When is a decoder better than Unmarshal?
Answer
For a large stream of many values
It keeps one value in memory at a time.
-
What does a truncated document produce?
Answer
An error, not io.EOF
EOF only comes between values.
-
What does a handler decode its body with?
Answer
json.NewDecoder(r.Body).Decode
The body is a stream and is read once.
The code on this page was run with go 1.26.10 on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev