Charsets and UTF-8

A String is a sequence of characters, and a file is a sequence of bytes. The charset is the rule that turns one into the other, and every read and every write uses one whether the program names it or not. This lesson shows the same text in three charsets, then reads it with the wrong one, which is the bug behind every café on the web.

The same text in three charsets

The text holds a Latin letter with an accent, a Cyrillic letter and a Chinese character. Each charset encodes it and the program prints the length and the hex:

import java.nio.charset.Charset; import java.nio.charset.StandardCharsets; import java.util.HexFormat; public class Utf8 { public static void main(String[] args) { String text = "café я 漢"; HexFormat hex = HexFormat.of(); for (Charset charset : new Charset[] {StandardCharsets.UTF_8, StandardCharsets.ISO_8859_1, StandardCharsets.US_ASCII}) { byte[] bytes = text.getBytes(charset); System.out.println(charset.name() + " " + bytes.length + " bytes " + hex.formatHex(bytes)); } System.out.println("characters: " + text.length()); System.out.println("code points: " + text.codePointCount(0, text.length())); System.out.println("utf-8 round trip: " + new String(text.getBytes(StandardCharsets.UTF_8), StandardCharsets.UTF_8).equals(text)); } }

javac -Xlint:all Utf8.java
java Utf8

UTF-8 12 bytes 636166c3a920d18f20e6bca2
ISO-8859-1 8 bytes 636166e9203f203f
US-ASCII 8 bytes 6361663f203f203f
characters: 8
code points: 8
utf-8 round trip: true

UTF-8 needs twelve bytes for eight characters: one byte for each of c, a, f and the space, two for the accented e (c3 a9), two for the Cyrillic letter and three for the Chinese one. Latin-1 has the accented e as one byte e9 and turns the other two into 3f, which is the question mark. ASCII cannot hold even the accented e, so it produces three question marks.

The question marks are the dangerous part: the encoding lost the information and the program was not told. A charset with no mapping for a character substitutes one and goes on, and the text that comes back is not the text that went in. The round trip through UTF-8 is the only one that returns the same string, which is why UTF-8 is the default for a new file or a new protocol.

The wrong charset read

The program writes a UTF-8 file and reads the bytes twice, once with the right charset and once with Latin-1:

import java.io.IOException; import java.nio.charset.StandardCharsets; import java.nio.file.Files; import java.nio.file.Path; public class Mojibake { public static void main(String[] args) throws IOException { Path file = Path.of("text.txt"); String text = "café"; Files.write(file, text.getBytes(StandardCharsets.UTF_8)); byte[] bytes = Files.readAllBytes(file); System.out.println("bytes: " + bytes.length); System.out.println("utf-8: [" + new String(bytes, StandardCharsets.UTF_8) + "]"); System.out.println("latin-1: [" + new String(bytes, StandardCharsets.ISO_8859_1) + "]"); System.out.println("latin-1 codes: " + codeList(new String(bytes, StandardCharsets.ISO_8859_1))); System.out.println("repaired: [" + new String(new String(bytes, StandardCharsets.ISO_8859_1).getBytes(StandardCharsets.ISO_8859_1), StandardCharsets.UTF_8) + "]"); System.out.println("same: " + new String(bytes, StandardCharsets.UTF_8).equals(text)); Files.delete(file); } static String codeList(String text) { StringBuilder builder = new StringBuilder(); for (char c : text.toCharArray()) { builder.append((int) c).append(' '); } return builder.toString().trim(); } }

javac -Xlint:all Mojibake.java
java Mojibake

bytes: 5
utf-8: [café]
latin-1: [café]
latin-1 codes: 99 97 102 195 169
repaired: [café]
same: true

The two bytes of the accented e, c3 and a9, are two valid Latin-1 characters, Ã and ©, so the wrong read does not fail: it produces a text that is plausible and wrong. That is mojibake, and its signature is the pair of characters in the third line. The code list shows the two as 195 and 169, which is the decimal form of the same two bytes.

Repairing a mojibake text

The repair is the inverse: encode the wrong text back into the charset it was wrongly read as, then decode the bytes with the right one. The fourth line does exactly that and gets café back. It works because Latin-1 maps every byte to one character and back with no loss, which is why it is the charset the mistake usually happens with.

A repair is a last resort: it works only when the text was read wrongly once and nothing else touched it. A file that was written with a lossy charset has the question marks already and no repair recovers the characters. The fix is at the source: name the charset on every read and write, and the question does not arise.

The default charset

A method without a charset uses the platform default, which is a property of the machine and the JVM. The program asks for it and for the names a program may meet:

import java.nio.charset.Charset; import java.nio.charset.StandardCharsets; import java.util.Set; public class Defaults { public static void main(String[] args) { System.out.println("default: " + Charset.defaultCharset().name()); System.out.println("utf-8 available: " + Charset.isSupported("UTF-8")); System.out.println("known charsets: " + Set.of("UTF-8", "US-ASCII", "ISO-8859-1").stream().filter(Charset::isSupported).count()); System.out.println("aliases for UTF-8: " + Charset.forName("UTF8").equals(StandardCharsets.UTF_8)); byte[] bytes = "hi".getBytes(StandardCharsets.UTF_8); System.out.println("decode: " + new String(bytes, StandardCharsets.UTF_8)); } }

javac -Xlint:all Defaults.java
java Defaults

default: UTF-8
utf-8 available: true
known charsets: 3
aliases for UTF-8: true
decode: hi

The default is UTF-8 in this container, and on a JDK 18 or later it is UTF-8 everywhere unless a flag says otherwise, which was not true before. A program that relies on it still differs from one that names it, because the reader of the code cannot tell which rule was used; naming the charset is a line of documentation as well as a decision.

Rules for text

Name the charset at every boundary: a file, a socket, a String built from bytes, a response body. Use UTF-8 unless the other side forces something else, and say in the code why when it does. Treat a String as text and a byte array as data, and convert between them only with a charset in hand. When a text arrives broken, find the boundary where the conversion happened rather than patching the characters.

The last point is the lesson of the third program: a repair that encodes and decodes again fixes a text only because the reader knows exactly which mistake was made. A program that logs the charset it used, and refuses to guess, is the one that does not need the repair.

Exercises

  1. Encode a text with an emoji in UTF-8 and count the bytes of the four-byte character.
  2. Write a file in UTF-16 and read it as UTF-8, and print both the code list and the text.
  3. Print the code point of each character of a text with codePoints.
  4. Repair the text of the second program by hand and compare it with the fourth line.

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. How many bytes does UTF-8 use for an accented Latin letter?

    Answer

    Two

    The letters up to U+007F take one byte, and the accented Latin ones take two.

  2. What happens when a charset cannot encode a character?

    Answer

    It writes a substitute, usually a question mark, and the information is lost

    The encoder replaces the unmappable character by default, which is why the ASCII line has question marks.

  3. What is mojibake?

    Answer

    Text decoded with a charset other than the one it was encoded with

    The bytes are read as characters of another charset, so the text shows the wrong ones.

  4. What does this program print? import java.nio.charset.StandardCharsets; public class Q1 { public static void main(String[] args) { String text = "é"; System.out.println(text.getBytes(StandardCharsets.UTF_8).length); System.out.println(text.getBytes(StandardCharsets.US_ASCII).length); System.out.println(new String(text.getBytes(StandardCharsets.US_ASCII), StandardCharsets.US_ASCII)); } }

    Answer

    2 / 1 / ?

    UTF-8 needs two bytes, ASCII substitutes one question mark, and the decode gives that mark.

  5. Why is the default charset a risk?

    Answer

    It depends on the machine and the JVM, so the same code reads text differently

    A program that relies on it behaves differently on two machines, and the code does not say which rule it used.

  6. What does a round trip through UTF-8 guarantee for any text?

    Answer

    The same string comes back when the right charset is used both ways

    UTF-8 can encode every character, which is why the same string comes back.

  7. What is the signature of a text read with the wrong charset?

    Answer

    A pair of characters such as the accented letters of Latin-1

    The bytes of a multi-byte character are valid single-byte characters of the other charset.

  8. What does this program print? import java.nio.charset.StandardCharsets; public class Q2 { public static void main(String[] args) { byte[] bytes = {99, 97, 102, (byte) 195, (byte) 169}; System.out.println(new String(bytes, StandardCharsets.UTF_8)); System.out.println(new String(bytes, StandardCharsets.ISO_8859_1).length()); } }

    Answer

    café / 5

    The five bytes are one e-acute in UTF-8 and five characters in Latin-1.

  9. Why does the repair work only for a single wrong read?

    Answer

    It reverses the exact mistake, and a second conversion or a lost character cannot be recovered

    Encoding back is the inverse of the one wrong decode, so the text must not have been changed since.

  10. Which charset does a new text file use?

    Answer

    UTF-8, named explicitly in the call

    UTF-8 holds every character, and naming it makes the file readable by any other program.

The code on this page was built with OpenJDK 21.0.13 (javac and java) and run on Alpine Linux in a Docker container.

Article author: Arthur Isaev