Binary files and streams

Bytes and characters

A Ruby string is bytes plus an encoding. length counts characters and bytesize counts bytes, and the two disagree as soon as the text leaves ASCII:

ruby -e 's = "café"; p s.bytesize, s.length, s.encoding'

5
4
#<Encoding:UTF-8>

Four characters, five bytes, because é takes two bytes in UTF-8. Any code that reads a fixed number of bytes and expects a number of characters has to convert: read bytes, then force_encoding and check valid_encoding?.

A string read from a binary file has the encoding ASCII-8BIT, which is Ruby's name for "just bytes":

ruby -e 'File.binwrite("b.bin", "\x00\x01".b); p File.binread("b.bin").encoding'

#<Encoding:BINARY (ASCII-8BIT)>

binread and binwrite

File.binread and File.binwrite are the same as read and write with binary mode, and .b on a literal marks the string as bytes. The bytes come back as integers:

ruby -e 'File.binwrite("b.bin", "\x00\x01".b); p File.binread("b.bin").bytes'

[0, 1]

Reading a PNG, a gzip or a small database this way is fine when the file is small. On a text file the binary form is wrong: the newline translation and the encoding both change, and the bytes are no longer what a text reader expects.

Reading in chunks

A large binary file is read a block at a time. read(n) returns n bytes and nil at the end, which makes it the condition of a loop:

File.open("b.bin", "rb") do |file| while (chunk = file.read(1)) p chunk.bytes end end

ruby -e 'File.binwrite("b.bin", "\x00\x01".b); File.open("b.bin", "rb") { |f| while (chunk = f.read(1)); p chunk.bytes; end }'

[0]
[1]

The chunk size is the tuning knob: 64 KB is a good default for a file, and a copy loop picks the size that keeps the disk busy without holding the whole file. The "rb" matters on Windows, where a text-mode handle would translate the bytes.

IO.copy_stream

To copy a file, do not read it and write it back in Ruby. IO.copy_stream moves the bytes in the interpreter's own loop, without building a string of the whole file:

ruby -e 'File.binwrite("b.bin", "\x00\x01".b); IO.copy_stream("b.bin", "c.bin"); p File.size("c.bin")'

It also takes streams rather than names, so the same call copies a network socket into a file or one open file into another. For a download, pass the response body and the target handle and skip the buffer entirely.

What to watch for

puts on a binary string can truncate at a null byte in a terminal, and a text editor may rewrite the line endings. Write binary data with binwrite or a "wb" handle, never with puts.

An encoding error such as invalid byte sequence in UTF-8 means a text operation met binary data. Find the read that should have been binread rather than forcing the encoding on the result.

Next: File metadata and permissions

Article author: Arthur Isaev