UTF-8, runes and the strings package
UTF-8 is the encoding Go assumes when it treats bytes as text: a code point takes one to four bytes, and the leading byte says how many. The compiler writes string literals as UTF-8, the standard library reads them that way, and unicode/utf8 is the package that answers questions about the encoding without allocating. The previous lesson counted bytes; this one counts letters.
Counting runes
utf8.RuneCountInString walks the string once and counts decodings, which is O(n) but allocates nothing. utf8.ValidString reports whether every byte sequence decodes as UTF-8, which is the check to run on input from the network or a file before it is trusted as text.
package main
import (
"fmt"
"unicode/utf8"
)
func main() {
for _, s := range []string{"hello", "héllo", "Привет", ""} {
fmt.Printf("%-8q bytes %d runes %d valid %v\n",
s, len(s), utf8.RuneCountInString(s), utf8.ValidString(s))
}
}
go run count.go
"hello" bytes 5 runes 5 valid true
"héllo" bytes 6 runes 5 valid true
"Привет" bytes 12 runes 6 valid true
"" bytes 0 runes 0 valid true
A Cyrillic letter takes two bytes and a CJK character three, so the byte count grows much faster than the rune count. The empty string is valid UTF-8, and its two counts are zero. Note that the table aligns itself through the width in %-8q, a verb the next lesson takes apart.
Decoding byte by byte
The loop that advances by the returned size is the manual form of range, and it is what a parser needs when it must know where each character starts. The index is always the start of a rune, and the size is one to four bytes. utf8.RuneLen gives the width of a rune without encoding it.
package main
import (
"fmt"
"unicode/utf8"
)
func main() {
s := "aé世"
for i := 0; i < len(s); {
r, size := utf8.DecodeRuneInString(s[i:])
fmt.Printf("at %d: %U %q width %d\n", i, r, r, size)
i += size
}
}
go run decode.go
at 0: U+0061 'a' width 1
at 1: U+00E9 'é' width 2
at 3: U+4E16 '世' width 3
The offsets 0, 1 and 3 follow the widths: one byte for a, two for é, three for 世. A loop that increments the index by one instead would try to decode the second byte of é on its own, get RuneError and then carry on inside the character.
Bytes that are not UTF-8
Decoding never fails: an invalid byte decodes as U+FFFD, the replacement character, with size one, so a loop always makes progress. That is a deliberate design choice of the encoding, and it means a program must check ValidString itself when it cares. A literal 0xff cannot appear in a Go source file, so the byte slice builds it.
package main
import (
"fmt"
"unicode/utf8"
)
func main() {
broken := string([]byte{'o', 'k', 0xff, '!'})
fmt.Println("bytes:", len(broken), "valid:", utf8.ValidString(broken))
r, size := utf8.DecodeRuneInString(broken[2:])
fmt.Printf("decoded %q size %d is RuneError: %v\n", r, size, r == utf8.RuneError)
for i, r := range broken {
fmt.Printf("%d:%q ", i, r)
}
fmt.Println()
}
go run invalid.go
bytes: 4 valid: false
decoded '�' size 1 is RuneError: true
0:'o' 1:'k' 2:'�' 3:'!'
The range printed '\ufffd' for the broken byte and continued with the exclamation mark, and the size of one proves that only that byte was consumed. Writing such a string back out preserves the bad byte, because a Go string is bytes first and text second.
The strings package
strings.Fields splits on any run of spaces, which is what parsing a line of text needs; Split splits on one exact separator and keeps the empty pieces. Cut returns the part before, the part after and whether the separator was there, which replaced the three-line Index pattern. None of them modify the input: they return substrings that share its bytes.
package main
import (
"fmt"
"strings"
)
func main() {
line := " GET /users/42 "
fields := strings.Fields(line)
fmt.Println("fields:", fields, len(fields))
method, rest, found := strings.Cut(fields[0], " ")
fmt.Println("cut:", method, rest, found)
path := strings.TrimPrefix(fields[1], "/")
parts := strings.Split(path, "/")
fmt.Println("parts:", parts, "joined:", strings.Join(parts, "."))
fmt.Println("upper:", strings.ToUpper("straße"))
fmt.Println("equal fold:", strings.EqualFold("GO", "go"), strings.EqualFold("go", "Go!"))
}
go run strings.go
fields: [GET /users/42] 2
cut: GET false
parts: [users 42] joined: users.42
upper: STRAßE
equal fold: true false
ToUpper("straße") gives STRASSE, because the German sharp s uppercases to two letters, so the result can be longer than the input and the function cannot be a byte-for-byte map. EqualFold compares under Unicode case folding and is the right way to compare a protocol keyword.
One rune, several bytes
The width of a rune is the number of bytes its UTF-8 encoding takes, and it has nothing to do with how wide the character looks on a terminal. A CJK ideograph is three bytes and two columns, an emoji is four bytes and two columns, and a combining accent adds bytes without adding a column at all. A table that must line up needs the column width from a library such as golang.org/x/text/width, not the byte count.
package main
import (
"fmt"
"unicode/utf8"
)
func main() {
for _, r := range "aé世😀" {
fmt.Printf("%q width %d utf8 bytes %d\n", r, utf8.RuneLen(r), len(string(r)))
}
}
go run width.go
'a' width 1 utf8 bytes 1
'é' width 2 utf8 bytes 2
'世' width 3 utf8 bytes 3
'😀' width 4 utf8 bytes 4
The ASCII letter takes one byte, é two, 世 three and the emoji four; that is the whole rule. Ranging over the literal decodes it one rune at a time, and len(string(r)) is the same number utf8.RuneLen reports, which is a useful cross-check when an encoding question comes up in a bug report.
Exercises
- Write a function that returns the longest rune in a string and the byte offset where it starts, using utf8.DecodeRuneInString in a loop.
- Read a comma-separated line with strings.Cut in a loop and report the fields in order. What happens to empty fields at the end of the line?
- Compare strings.ToLower(s) == "go" with strings.EqualFold(s, "go") for the input "GO" and for "Go!". Which one should a command parser use?
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
Which function counts characters rather than bytes?
Answer
utf8.RuneCountInString
RuneCountInString decodes the string and counts runes; len counts the bytes.
-
What does this program print for the second line, the string héllo?
package main import ( "fmt" "unicode/utf8" ) func main() { for _, s := range []string{"hello", "héllo", "Привет", ""} { fmt.Printf("%-8q bytes %d runes %d valid %v\n", s, len(s), utf8.RuneCountInString(s), utf8.ValidString(s)) } }Answer
"hello" bytes 5 runes 5 valid true / "héllo" bytes 6 runes 5 valid true / "Привет" bytes 12 runes 6 valid true / "" bytes 0 runes 0 valid true
é takes two bytes, so six bytes hold five runes and the string is valid UTF-8.
-
What does utf8.DecodeRuneInString return for an invalid byte?
Answer
RuneError and a size of one
Decoding cannot fail: the replacement character is returned with size one so a loop progresses.
-
What does this program print?
package main import ( "fmt" "unicode/utf8" ) func main() { s := "aé世" for i := 0; i < len(s); { r, size := utf8.DecodeRuneInString(s[i:]) fmt.Printf("at %d: %U %q width %d\n", i, r, r, size) i += size } }Answer
at 0: U+0061 'a' width 1 / at 1: U+00E9 'é' width 2 / at 3: U+4E16 '世' width 3
The next offset is the previous one plus the width: 0 plus 1, then 1 plus 2.
-
What does this program print?
package main import ( "fmt" "unicode/utf8" ) func main() { broken := string([]byte{'o', 'k', 0xff, '!'}) fmt.Println("bytes:", len(broken), "valid:", utf8.ValidString(broken)) r, size := utf8.DecodeRuneInString(broken[2:]) fmt.Printf("decoded %q size %d is RuneError: %v\n", r, size, r == utf8.RuneError) for i, r := range broken { fmt.Printf("%d:%q ", i, r) } fmt.Println() }Answer
bytes: 4 valid: false / decoded '�' size 1 is RuneError: true / 0:'o' 1:'k' 2:'�' 3:'!'
0xff is not a valid UTF-8 byte: ValidString is false and the decoder consumes one byte.
-
What is the difference between strings.Fields and strings.Split?
Answer
Fields splits on any run of white space, Split on one exact separator
Fields is for tokenising a line; Split is for a known delimiter and keeps empty pieces.
-
What does the line about the fields print?
package main import ( "fmt" "strings" ) func main() { line := " GET /users/42 " fields := strings.Fields(line) fmt.Println("fields:", fields, len(fields)) method, rest, found := strings.Cut(fields[0], " ") fmt.Println("cut:", method, rest, found) path := strings.TrimPrefix(fields[1], "/") parts := strings.Split(path, "/") fmt.Println("parts:", parts, "joined:", strings.Join(parts, ".")) fmt.Println("upper:", strings.ToUpper("straße")) fmt.Println("equal fold:", strings.EqualFold("GO", "go"), strings.EqualFold("go", "Go!")) }Answer
fields: [GET /users/42] 2 / cut: GET false / parts: [users 42] joined: users.42 / upper: STRAßE / equal fold: true false
Fields drops the surrounding spaces and splits on the one run between GET and the path.
-
Why can strings.ToUpper return a string longer than its input?
Answer
Some code points uppercase to several characters, such as ß to SS
Case mapping is defined on runes, and one rune may map to several.
-
What does utf8.RuneLen measure?
Answer
The bytes the UTF-8 encoding of a rune takes
RuneLen is about the encoding, not about rendering; column width is another question.
-
Which comparison treats GO and go as equal on purpose?
Answer
strings.EqualFold
EqualFold compares under Unicode case folding, which is what protocol keywords need.
The code on this page was run with go 1.26.10 on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev