bufio: buffered reading and writing

A reader returns bytes, and the operating system charges for every call. bufio reads a large block once and hands out small pieces, or collects small writes and sends them in one call. The package also turns a reader into a line scanner, which is why most Go programs read text through bufio rather than through Read directly. The previous lesson made the interfaces; this one wraps them.

bufio.Scanner

Scanner reads a token at a time, a line by default, and returns it as a string without the newline. Scan reports whether there was a token, so the loop ends at EOF, and Err reports a read error after the loop. The scanner owns no buffer of yours: it reuses one, so Text is only valid until the next Scan.

package main import ( "bufio" "fmt" "strings" ) func main() { scanner := bufio.NewScanner(strings.NewReader("first line\nsecond line\nthird line\n")) line := 0 for scanner.Scan() { line++ fmt.Printf("%d: %s\n", line, scanner.Text()) } fmt.Println("error:", scanner.Err()) }

go build -o app main.go && ./app

1: first line
2: second line
3: third line
error: <nil>

The newline is gone from every line, including the last, and the loop ran three times. A file whose last line has no newline still yields that line; a file with a very long line fails with bufio.Scanner: token too long unless Buffer raised the limit.

bufio.Reader

ReadString takes a delimiter and returns everything up to and including it. The error arrives with the last piece, so the loop prints the piece and then breaks: the final line of a file often comes with io.EOF and no delimiter. The reader can also return a short piece with a nil error, and the caller keeps reading.

package main import ( "bufio" "fmt" "strings" ) func main() { reader := bufio.NewReader(strings.NewReader("hello\nworld\n")) for { line, err := reader.ReadString('\n') if line != "" { fmt.Printf("read %d bytes: %q\n", len(line), line) } if err != nil { break } } fmt.Println("done") }

go build -o app main.go && ./app

read 6 bytes: "hello\n"
read 6 bytes: "world\n"
done

Six bytes include the newline, which %q shows. ReadString allocates a new string for each call, while Scanner reuses its buffer; a program that keeps every line must copy or convert anyway, which is why the scanner is the default for reading and ReadString is for a header or one line.

bufio.Writer and Flush

A bufio.Writer collects writes in memory and sends them to the underlying writer when the buffer is full or when Flush is called. The buffer below is still empty after three Fprintf calls and holds twenty-one bytes after the flush: a program that forgets the flush loses the tail, and a deferred flush is the habit that prevents it.

package main import ( "bufio" "bytes" "fmt" ) func main() { var buf bytes.Buffer writer := bufio.NewWriter(&buf) for i := 1; i <= 3; i++ { fmt.Fprintf(writer, "line %d\n", i) } fmt.Println("before flush:", buf.Len()) if err := writer.Flush(); err != nil { fmt.Println("flush error:", err) } fmt.Println("after flush:", buf.Len()) fmt.Print(buf.String()) }

go build -o app main.go && ./app

before flush: 0
after flush: 21
line 1
line 2
line 3

Three lines of seven bytes are twenty-one, and none of them reached the buffer until the flush. The default buffer is 4096 bytes, which is the right size for a file and the wrong one for a program that must write a line before it waits for input; os.Stdout is unbuffered for that reason.

ScanWords and Split

The scanner's token is decided by its split function. bufio.ScanWords returns words, ScanRunes returns runes and ScanBytes bytes, and a program can supply its own function with the same signature. The sentence below is split into five words with no regular expression and no allocation of a slice.

package main import ( "bufio" "fmt" "strings" ) func main() { scanner := bufio.NewScanner(strings.NewReader("the quick brown fox jumps")) scanner.Split(bufio.ScanWords) words := 0 for scanner.Scan() { words++ fmt.Println("word:", scanner.Text()) } fmt.Println("total:", words) }

go build -o app main.go && ./app

word: the
word: quick
word: brown
word: fox
word: jumps
total: 5

A split function returns the token, its length and an error, and the scanner calls it until the input ends. That is how a program counts words in a large file without loading it: the scanner reads its buffer and the split function cuts it, and the memory stays at the size of the buffer.

When to buffer

Buffer when a program makes many small calls on the same reader or writer: a log line per request, a line per record, a byte at a time. Do not buffer when the program must see each write at once, as a prompt on a terminal or a protocol reply; an unbuffered writer is then the correct choice. The buffer size is a trade between syscalls and latency, and the default of 4096 bytes is a good start for both directions.

Two wrappers compose with this lesson: bufio.NewReaderSize and NewWriterSize take the size, and a Scanner over a bufio.Reader reads through the same buffer. The next lesson opens files, and the scanner is the tool that reads them line by line.

Exercises

  1. Read the same input with Scanner and with ReadString and compare the bytes of the last line. Which one keeps the newline?
  2. Remove the Flush from the writer program and print the buffer. What is lost, and what does the program print?
  3. Write a split function that returns comma-separated fields and use it on "a,b,c". Which method of the scanner gives the field?

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. What does bufio.Scanner return by default?

    Answer

    Lines without the newline

    ScanLines is the default split function.

  2. How does the scanner loop end?

    Answer

    Scan returns false at EOF or on an error

    Err then reports a read error, if any.

  3. Where is the newline after ReadString?

    Answer

    It is included in the returned string

    The delimiter is part of the result.

  4. What does bufio.Writer do with small writes?

    Answer

    Collects them until the buffer is full or Flush is called

    That is the point of buffering.

  5. What happens if a program forgets Flush?

    Answer

    The tail of the output is lost

    A deferred Flush prevents it.

  6. Which split function returns words?

    Answer

    bufio.ScanWords

    It splits on white space.

  7. Why does a scanner not need a slice of all lines?

    Answer

    It reuses one buffer and returns a token at a time

    Text is valid until the next Scan.

  8. What is the default buffer size of bufio?

    Answer

    4096 bytes

    NewReaderSize and NewWriterSize change it.

  9. When should a program not buffer its output?

    Answer

    When each write must be visible at once, such as a prompt

    A terminal prompt must appear before the input.

  10. What does Scanner.Buffer change?

    Answer

    The largest token the scanner will accept

    The default limit is 64 kilobytes per token.

The code on this page was run with go 1.26.10 on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev