Unicode, bytes and encodings

Bytes against characters

A Ruby string is a sequence of bytes and an encoding that says how to read them. length counts characters, bytesize counts bytes, and the two differ as soon as the text leaves ASCII:

ruby -e 's = "café"; p s.bytesize, s.length, s.encoding'

5 4 #<Encoding:UTF-8>

Five bytes for four characters: é is two bytes in UTF-8. A database column sized in bytes, a terminal width counted in characters and a truncation that cuts a string at a byte position are the three places this distinction shows up as a bug.

force_encoding

A string read from a binary file is tagged ASCII-8BIT, which means "bytes, encoding unknown". force_encoding reinterprets the same bytes as another encoding without converting them:

ruby -e 's = "caf\xC3\xA9".b; p s.encoding; p s.valid_encoding?; u = s.force_encoding("UTF-8"); p u.valid_encoding?; p u.length'

#<Encoding:BINARY (ASCII-8BIT)> true true 4

The binary string was valid as bytes; the UTF-8 reading of the same bytes is valid too, and now length counts four characters. force_encoding is the right tool when the bytes are known to be UTF-8 and the tag was lost, not when the bytes are actually broken.

scrub

A truncated file leaves an incomplete sequence. The string is not valid UTF-8, and any character operation on it can raise; scrub replaces the broken bytes and returns a usable string:

ruby -e 'bad = "caf\xC3".b.force_encoding("UTF-8"); p bad.valid_encoding?; p bad.scrub'

false "caf�"

scrub takes a replacement string, and scrub! changes the receiver in place. valid_encoding? is the cheap test to run before a parse: a JSON parser that meets broken UTF-8 fails with a less helpful message than the one line that says which file is broken.

Transcoding

encode converts characters to another encoding. The bytes change, and the length in bytes changes with them:

ruby -e 'p "café".encode("UTF-16LE").bytes'

[99, 0, 97, 0, 102, 0, 233, 0]

Every character is two bytes now, and the é is a single code unit rather than two UTF-8 bytes. An invalid input raises instead of writing a corrupt file:

ruby -e '"caf\xC3".b.force_encoding("UTF-8").encode("UTF-16LE")'

-e:1:in 'String#encode': incomplete "\xC3" on UTF-8 (Encoding::InvalidByteSequenceError)

encode("UTF-16LE", invalid: :replace, undef: :replace) writes a replacement instead of raising, which is what a converter that must not lose the rest of the file needs. The two options are separate: one for an invalid sequence in the source, one for a character the target cannot represent.

What to watch for

Declare the encoding on the read: File.read(path, encoding: "UTF-8"). The default external encoding depends on the machine, so a file that reads on a laptop can fail on a server.

A regex and a slice work on characters, a socket and a file offset work on bytes. Mixing the two is how a multi-byte character gets cut in half, and the fix is to work in characters until the point where bytes are actually needed.

Next: JSON and YAML

Article author: Arthur Isaev