bufio: buffered reading and writing
A reader returns bytes, and the operating system charges for every call. bufio reads a large block once and hands out small pieces, or collects small writes and sends them in one call. The package also turns a reader into a line scanner, which is why most Go programs read text through bufio rather than through Read directly. The previous lesson made the interfaces; this one wraps them.
bufio.Scanner
Scanner reads a token at a time, a line by default, and returns it as a string without the newline. Scan reports whether there was a token, so the loop ends at EOF, and Err reports a read error after the loop. The scanner owns no buffer of yours: it reuses one, so Text is only valid until the next Scan.
package main
import (
"bufio"
"fmt"
"strings"
)
func main() {
scanner := bufio.NewScanner(strings.NewReader("first line\nsecond line\nthird line\n"))
line := 0
for scanner.Scan() {
line++
fmt.Printf("%d: %s\n", line, scanner.Text())
}
fmt.Println("error:", scanner.Err())
}
go build -o app main.go && ./app
1: first line
2: second line
3: third line
error: <nil>
The newline is gone from every line, including the last, and the loop ran three times. A file whose last line has no newline still yields that line; a file with a very long line fails with bufio.Scanner: token too long unless Buffer raised the limit.
bufio.Reader
ReadString takes a delimiter and returns everything up to and including it. The error arrives with the last piece, so the loop prints the piece and then breaks: the final line of a file often comes with io.EOF and no delimiter. The reader can also return a short piece with a nil error, and the caller keeps reading.
package main
import (
"bufio"
"fmt"
"strings"
)
func main() {
reader := bufio.NewReader(strings.NewReader("hello\nworld\n"))
for {
line, err := reader.ReadString('\n')
if line != "" {
fmt.Printf("read %d bytes: %q\n", len(line), line)
}
if err != nil {
break
}
}
fmt.Println("done")
}
go build -o app main.go && ./app
read 6 bytes: "hello\n"
read 6 bytes: "world\n"
done
Six bytes include the newline, which %q shows. ReadString allocates a new string for each call, while Scanner reuses its buffer; a program that keeps every line must copy or convert anyway, which is why the scanner is the default for reading and ReadString is for a header or one line.
bufio.Writer and Flush
A bufio.Writer collects writes in memory and sends them to the underlying writer when the buffer is full or when Flush is called. The buffer below is still empty after three Fprintf calls and holds twenty-one bytes after the flush: a program that forgets the flush loses the tail, and a deferred flush is the habit that prevents it.
package main
import (
"bufio"
"bytes"
"fmt"
)
func main() {
var buf bytes.Buffer
writer := bufio.NewWriter(&buf)
for i := 1; i <= 3; i++ {
fmt.Fprintf(writer, "line %d\n", i)
}
fmt.Println("before flush:", buf.Len())
if err := writer.Flush(); err != nil {
fmt.Println("flush error:", err)
}
fmt.Println("after flush:", buf.Len())
fmt.Print(buf.String())
}
go build -o app main.go && ./app
before flush: 0
after flush: 21
line 1
line 2
line 3
Three lines of seven bytes are twenty-one, and none of them reached the buffer until the flush. The default buffer is 4096 bytes, which is the right size for a file and the wrong one for a program that must write a line before it waits for input; os.Stdout is unbuffered for that reason.
ScanWords and Split
The scanner's token is decided by its split function. bufio.ScanWords returns words, ScanRunes returns runes and ScanBytes bytes, and a program can supply its own function with the same signature. The sentence below is split into five words with no regular expression and no allocation of a slice.
package main
import (
"bufio"
"fmt"
"strings"
)
func main() {
scanner := bufio.NewScanner(strings.NewReader("the quick brown fox jumps"))
scanner.Split(bufio.ScanWords)
words := 0
for scanner.Scan() {
words++
fmt.Println("word:", scanner.Text())
}
fmt.Println("total:", words)
}
go build -o app main.go && ./app
word: the
word: quick
word: brown
word: fox
word: jumps
total: 5
A split function returns the token, its length and an error, and the scanner calls it until the input ends. That is how a program counts words in a large file without loading it: the scanner reads its buffer and the split function cuts it, and the memory stays at the size of the buffer.
When to buffer
Buffer when a program makes many small calls on the same reader or writer: a log line per request, a line per record, a byte at a time. Do not buffer when the program must see each write at once, as a prompt on a terminal or a protocol reply; an unbuffered writer is then the correct choice. The buffer size is a trade between syscalls and latency, and the default of 4096 bytes is a good start for both directions.
Two wrappers compose with this lesson: bufio.NewReaderSize and NewWriterSize take the size, and a Scanner over a bufio.Reader reads through the same buffer. The next lesson opens files, and the scanner is the tool that reads them line by line.
Exercises
- Read the same input with Scanner and with ReadString and compare the bytes of the last line. Which one keeps the newline?
- Remove the Flush from the writer program and print the buffer. What is lost, and what does the program print?
- Write a split function that returns comma-separated fields and use it on "a,b,c". Which method of the scanner gives the field?
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
What does bufio.Scanner return by default?
Answer
Lines without the newline
ScanLines is the default split function.
-
How does the scanner loop end?
Answer
Scan returns false at EOF or on an error
Err then reports a read error, if any.
-
Where is the newline after ReadString?
Answer
It is included in the returned string
The delimiter is part of the result.
-
What does bufio.Writer do with small writes?
Answer
Collects them until the buffer is full or Flush is called
That is the point of buffering.
-
What happens if a program forgets Flush?
Answer
The tail of the output is lost
A deferred Flush prevents it.
-
Which split function returns words?
Answer
bufio.ScanWords
It splits on white space.
-
Why does a scanner not need a slice of all lines?
Answer
It reuses one buffer and returns a token at a time
Text is valid until the next Scan.
-
What is the default buffer size of bufio?
Answer
4096 bytes
NewReaderSize and NewWriterSize change it.
-
When should a program not buffer its output?
Answer
When each write must be visible at once, such as a prompt
A terminal prompt must appear before the input.
-
What does Scanner.Buffer change?
Answer
The largest token the scanner will accept
The default limit is 64 kilobytes per token.
The code on this page was run with go 1.26.10 on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev