How encoding/json uses reflection

encoding/json is a reflection walk over a type with a cache in front of it. The walk reads the exported fields, the tags and the kinds, and the cache keeps the plan per type so the second message of a million costs a map lookup and a fill. A custom method can replace the walk for one type, which is how a value with a unit or a time in a special layout is encoded.

The field walk

Marshal reads every exported field, takes the name from the json tag and skips a field whose tag is a dash. Unmarshal matches the keys to the same names and ignores an unknown key or a field it cannot see. The struct below has a dash field, an unexported field and a key that does not exist, and only the two real fields survive the round trip.

package main import ( "encoding/json" "fmt" ) type User struct { Name string `json:"name"` Age int `json:"age"` Secret string `json:"-"` private string } func main() { u := User{Name: "ada", Age: 36, Secret: "hidden", private: "unexported"} data, err := json.Marshal(u) fmt.Println("marshal:", string(data)) fmt.Println("error:", err) var back User err = json.Unmarshal([]byte(`{"name":"grace","age":45,"secret":"ignored"}`), &back) fmt.Printf("unmarshal: %+v\n", back) fmt.Println("error:", err) }

go build -o app main.go && ./app

marshal: {"name":"ada","age":36}
error: <nil>
unmarshal: {Name:grace Age:45 Secret: private:}
error: <nil>

The marshal output has two keys, because Secret is a dash and private is unexported. The unmarshal filled the name and the age and left the other two empty; the unknown secret key was ignored. That is the whole behaviour of the package on a struct, and the rest is the tags and the kinds.

A custom marshaller

A type with a MarshalJSON method replaces the walk for itself. The method returns the bytes it wants, and the package trusts them: it does not check that they are valid JSON. The temperature below writes a string with a unit, which is a value a client can read without knowing the struct.

package main import ( "encoding/json" "fmt" ) type Temperature float64 func (t Temperature) MarshalJSON() ([]byte, error) { return []byte(fmt.Sprintf("%q", fmt.Sprintf("%.1f C", float64(t)))), nil } func main() { data, err := json.Marshal(struct { Temp Temperature `json:"temp"` }{Temp: 21.5}) fmt.Println(string(data)) fmt.Println("error:", err) }

go build -o app main.go && ./app

{"temp":"21.5 C"}
error: <nil>

The struct's walk reached the field and called the method, and the result is a JSON string. The method must return an error for a value it cannot represent, and a panic inside it becomes an error from Marshal, not a crash: the package recovers and reports json: error calling MarshalJSON.

A custom unmarshaller

UnmarshalJSON gets the raw bytes of the value and writes through a pointer. The method below reads a string, strips the unit and parses the number, so the same wire value round-trips through both custom methods. A decoder that finds the method does not walk the type at all.

package main import ( "encoding/json" "fmt" "strconv" "strings" ) type Temperature float64 func (t *Temperature) UnmarshalJSON(data []byte) error { var s string if err := json.Unmarshal(data, &s); err != nil { return err } s = strings.TrimSuffix(s, " C") value, err := strconv.ParseFloat(s, 64) if err != nil { return err } *t = Temperature(value) return nil } func main() { var v struct { Temp Temperature `json:"temp"` } err := json.Unmarshal([]byte(`{"temp":"21.5 C"}`), &v) fmt.Println("temp:", float64(v.Temp)) fmt.Println("error:", err) }

go build -o app main.go && ./app

temp: 21.5
error: <nil>

The value arrived as a string and the field holds the number at the end. A method that returns an error makes Unmarshal return it with the field path, which is how a client learns which value was wrong. The pair is the documented way to give a type a wire format of its own without a separate DTO.

The walk by hand

The function below is the small core of the package: reflect over the type, read the tag, switch on the kind, write the key and the value. Its output matches json.Marshal for this struct, which is the point: the package is not magic, it is this loop with a cache and a lot of edge cases.

package main import ( "encoding/json" "fmt" "reflect" "strings" ) type Note struct { ID int `json:"id"` Title string `json:"title"` Note string `json:"-"` } func encode(v any) string { rv := reflect.ValueOf(v) rt := rv.Type() var parts []string for i := 0; i < rt.NumField(); i++ { field := rt.Field(i) tag := field.Tag.Get("json") if tag == "-" { continue } name := field.Name if tag != "" { name = strings.Split(tag, ",")[0] } value := rv.Field(i) if value.Kind() == reflect.Int { parts = append(parts, fmt.Sprintf("%q:%d", name, value.Int())) } else { parts = append(parts, fmt.Sprintf("%q:%q", name, value.String())) } } return "{" + strings.Join(parts, ",") + "}" } func main() { n := Note{ID: 1, Title: "buy milk", Note: "ignored"} expected, err := json.Marshal(n) if err != nil { panic(err) } fmt.Println("encoding/json:", string(expected)) fmt.Println("reflection: ", encode(n)) }

go build -o app main.go && ./app

encoding/json: {"id":1,"title":"buy milk"}
reflection: {"id":1,"title":"buy milk"}

The two lines are the same, and the hand-written version already has a bug the package does not: a string with a quote is not escaped. That is the lesson of re-implementing it: the loop is short and the escaping table, the float formats and the recursion are long. The package caches the plan per type, which is why the real one is fast enough to use by default.

A field deep in a slice

The error path names the Go type, the field and the type it expected, and for a nested value it names the outer field too. The message below says Item.items.age: the field age of an Item inside the items array. A client that logs the message can find the value without a debugger.

package main import ( "encoding/json" "fmt" ) type Item struct { Age int `json:"age"` } type Response struct { Items []Item `json:"items"` } func main() { var r Response err := json.Unmarshal([]byte(`{"items":[{"age":"old"}]}`), &r) fmt.Println("error:", err) }

go build -o app main.go && ./app

error: json: cannot unmarshal string into Go struct field Item.items.age of type int

The message is produced by the field walk after the decoder has descended into the slice, and it is the same text whether the value is at the top or three levels down. A handler maps it to a 400 and a test asserts on the field name, which is stable, rather than on the whole sentence.

What the package caches

The first marshal of a type builds a plan: the fields to include, their names, their order and the encoders for their kinds. The plan is cached per type, and every later value of that type reuses it. The cache is why a single struct type encodes at a steady speed and why a program that builds types at run time pays for each one. It is also why the tag of a field is a compatibility promise: the plan is made once per type and the keys are fixed by it.

The next lesson leaves the encoder for the memory model, the rules that say when one goroutine sees a write made by another. The reflection here is all on one goroutine; the next rules are about the ones that are not.

Exercises

  1. Add a MarshalJSON to a struct instead of a float type and read the output of a field that holds it. Which method wins, the struct's or the field walk?
  2. Remove the json tag from the Note struct and run the hand-written encoder. Which keys change, and does encoding/json agree?
  3. Give the custom unmarshaller a value without the unit and read the error. What does the message name, and where would a handler turn it into a 400?

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. Which fields does encoding/json marshal?

    Answer

    The exported fields, with the json tag names

    A dash tag skips one and unexported fields never take part.

  2. What does a dash in a json tag do on Unmarshal?

    Answer

    The field is ignored

    The key never matches.

  3. What does MarshalJSON replace?

    Answer

    The field walk for that type

    The method returns the bytes for the value.

  4. What does a panic inside MarshalJSON become?

    Answer

    An error from Marshal

    The package recovers and reports it.

  5. What does UnmarshalJSON receive?

    Answer

    The raw bytes of the value

    It parses what it needs and writes through the pointer.

  6. What does the hand-written encoder miss?

    Answer

    Escaping a string with a quote

    That is why the real package is longer.

  7. What does the nested error name?

    Answer

    The Go type, the outer field and the inner field

    It is produced after descending into the slice.

  8. What does encoding/json cache per type?

    Answer

    The field plan with names and encoders

    The first marshal builds it.

  9. Why is a tag a compatibility promise?

    Answer

    The cached plan fixes the keys for the type

    A rename changes the wire format.

  10. When is a custom marshaller the right tool?

    Answer

    When the wire format differs from the struct layout

    A unit or a special date is the classic case.

The code on this page was run with go 1.26.10 on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev