Bytes, code points and UTF-8
A std::string holds bytes, and UTF-8 is an encoding of code points into one to four bytes. For ASCII the two coincide, which is why so much code works until the first accented letter, the first emoji or the first Cyrillic name.
The lesson decodes UTF-8 by hand: it separates the bytes of a character, counts code points instead of bytes and encodes a code point back. Lesson 017 treated std::string as a sequence of chars; this lesson says which chars mean what.
Bytes and a literal
A char is one byte. A UTF-8 literal like "h\u00e9llo" is five code points and six bytes, because the e-acute takes two bytes. The compiler stores the bytes and the size tells the count.
The program prints the length of an ASCII word and of the same word with an accent. The second is longer by one byte, which is the whole subject of the lesson in one number.
The u8 prefix makes the encoding explicit in the type: the literal is a const char8_t array, and converting it to std::string takes a cast or the u8string type. The bytes are the same.
#include <iostream>
#include <string>
int main()
{
std::string plain = "hello";
std::string accented = "h\u00e9llo";
std::cout << plain.size() << ' ' << accented.size() << '\n';
std::cout << accented << '\n';
}
g++ -std=c++23 -Wall -Wextra bytes.cpp -o bytes
./bytes
5 6
héllo
Decoding a code point
UTF-8 marks the first byte of a character: a zero in the top bit means one byte, 110 means two, 1110 means three and 11110 means four. The continuation bytes start with 10 and carry six bits each.
The program's decoder reads those markers and rebuilds each code point from the bits. It prints the code point of every character of a mixed word, and the accented letter appears as 233.
This is what a library does for you, and writing it once makes the format concrete. The standard library's conversions live in <codecvt>, which is deprecated, and in the formatting and locale facilities; a small decoder like this is often the clearest option.
#include <iostream>
#include <string>
#include <vector>
std::vector<char32_t> decode(const std::string& text)
{
std::vector<char32_t> points;
for (std::size_t i = 0; i < text.size();) {
unsigned char lead = static_cast<unsigned char>(text[i]);
char32_t point = 0;
int extra = 0;
if ((lead & 0x80) == 0) {
point = lead;
} else if ((lead & 0xE0) == 0xC0) {
point = lead & 0x1F;
extra = 1;
} else if ((lead & 0xF0) == 0xE0) {
point = lead & 0x0F;
extra = 2;
} else {
point = lead & 0x07;
extra = 3;
}
++i;
for (int k = 0; k < extra; ++k, ++i) {
point = (point << 6) | (static_cast<unsigned char>(text[i]) & 0x3F);
}
points.push_back(point);
}
return points;
}
int main()
{
for (char32_t point : decode("h\u00e9llo")) {
std::cout << static_cast<unsigned long>(point) << ' ';
}
std::cout << '\n';
}
g++ -std=c++23 -Wall -Wextra decode.cpp -o decode
./decode
104 233 108 108 111
Counting characters
size() counts bytes, not characters. For a word with one accented letter the two differ by one, and for a string of Cyrillic they differ by a factor of two.
The program prints size() and the number of code points returned by the decoder of the previous section. The second number is the one a user would call the length.
A text editor, a column limit and a truncation all need the character count. Doing it with size() cuts a UTF-8 character in half, which the last section shows.
#include <iostream>
#include <string>
std::size_t code_points(const std::string& text)
{
std::size_t count = 0;
for (unsigned char byte : text) {
if ((byte & 0xC0) != 0x80) {
++count;
}
}
return count;
}
int main()
{
std::string text = "\u041f\u0440\u0438\u0432\u0435\u0442";
std::cout << text.size() << ' ' << code_points(text) << ' ' << text << '\n';
}
g++ -std=c++23 -Wall -Wextra count.cpp -o count
./count
12 6 Привет
Encoding a code point
Encoding is the reverse: a code point below 128 is one byte, below 2048 two, below 65536 three and the rest four. The program encodes three code points and prints the bytes in hexadecimal.
The output for 233 is c3 a9, the two bytes of e-acute in UTF-8, and the Euro sign 8364 becomes e2 82 ac. The last value is the Cyrillic capital Pe, which takes two bytes.
A std::string can hold the result because a byte is a byte. The program appends the bytes to a string and prints it, which is how a program builds UTF-8 text from code points.
#include <iostream>
#include <string>
std::string encode(char32_t point)
{
std::string out;
if (point < 0x80) {
out += static_cast<char>(point);
} else if (point < 0x800) {
out += static_cast<char>(0xC0 | (point >> 6));
out += static_cast<char>(0x80 | (point & 0x3F));
} else {
out += static_cast<char>(0xE0 | (point >> 12));
out += static_cast<char>(0x80 | ((point >> 6) & 0x3F));
out += static_cast<char>(0x80 | (point & 0x3F));
}
return out;
}
int main()
{
std::string text = encode(233) + encode(8364) + encode(1055);
std::cout << std::hex;
for (unsigned char byte : text) {
std::cout << static_cast<int>(byte) << ' ';
}
std::cout << std::dec << '\n' << text << '\n';
}
g++ -std=c++23 -Wall -Wextra encode.cpp -o encode
./encode
c3 a9 e2 82 ac d0 9f
é€П
Cutting a character in half
substr counts bytes, so substr(0, 2) of a word that starts with a two-byte letter takes the whole letter, while substr(0, 1) takes only its first byte and leaves an invalid sequence.
The program prints the first byte of the accented word as a number and then the two-byte prefix. The single byte is not a character on its own, and a program that truncates text this way produces broken UTF-8.
The fix is to cut at a code point boundary: walk the string, skip continuation bytes, and stop after the number of characters wanted. The counting loop of the third section is the start of that function.
#include <iostream>
#include <string>
int main()
{
std::string text = "h\u00e9llo";
std::string first_byte = text.substr(0, 2).substr(1, 1);
std::string two_bytes = text.substr(1, 2);
std::cout << std::hex << static_cast<int>(static_cast<unsigned char>(first_byte[0]))
<< ' ' << static_cast<int>(static_cast<unsigned char>(two_bytes[0]))
<< ' ' << static_cast<int>(static_cast<unsigned char>(two_bytes[1]))
<< std::dec << '\n';
}
g++ -std=c++23 -Wall -Wextra cut.cpp -o cut
./cut
c3 c3 a9
Exercises
- Print the code points of your name with the decoder and compare the count with size().
- Write the encoder's fourth case for code points above 65535 and test it with an emoji.
- Write a function that truncates a UTF-8 string to n characters without cutting a character.
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
What does a
std::stringhold?Answer
Bytes
The string is a sequence of chars, and the encoding decides what the bytes mean.
-
How many bytes does an ASCII character take in UTF-8?
Answer
One
The first 128 code points are one byte, which is why ASCII text works unchanged.
-
What does the first byte of a multi-byte UTF-8 sequence encode?
Answer
The number of bytes in the sequence
The high bits of the lead byte give the length, and the continuation bytes start with 10.
-
What happens when you cut a UTF-8 string in the middle of a character?
Answer
The bytes no longer decode to the original text
The continuation bytes are left without their lead byte, and the decoder reports an error.
-
What does this program print?
#include <iostream> #include <string> int main() { std::string text = "hello"; std::cout << text.size() << ' ' << std::string{"h\xc3\xa9llo"}.size(); }Answer
5 6
The accented letter takes two bytes, so the second string is one byte longer.
-
What does this program print?
#include <iostream> #include <string> int main() { std::string text = "h\xc3\xa9llo"; for (unsigned char byte : text) std::cout << static_cast<int>(byte) << ' '; }Answer
104 195 169 108 108 111
The accented letter is the two bytes 195 and 169, and the other letters are ASCII.
-
What does this program print?
#include <iostream> #include <string> int main() { std::string text = "\xd0\x9f\xd1\x80\xd0\xb8\xd0\xb2\xd0\xb5\xd1\x82"; int characters = 0; for (unsigned char byte : text) if ((byte & 0xC0) != 0x80) ++characters; std::cout << text.size() << ' ' << characters << ' ' << text; }Answer
12 6 Привет
Six Cyrillic letters take two bytes each, and the lead-byte count gives six characters.
-
What does this program print?
#include <iostream> int main() { unsigned code = 0x20AC; unsigned char bytes[3]; bytes[0] = static_cast<unsigned char>(0xE0 | (code >> 12)); bytes[1] = static_cast<unsigned char>(0x80 | ((code >> 6) & 0x3F)); bytes[2] = static_cast<unsigned char>(0x80 | (code & 0x3F)); std::cout << std::hex << (int)bytes[0] << ' ' << (int)bytes[1] << ' ' << (int)bytes[2] << ' '; std::cout.write(reinterpret_cast<char*>(bytes), 3); }Answer
e2 82 ac and the euro sign
The three bytes encode the euro sign, and the write prints them as the character.
-
What does this program print?
#include <iostream> #include <string> int main() { std::string text = "h\xc3\xa9llo"; std::cout << text.substr(1, 1).size() << ' '; std::cout << static_cast<int>(static_cast<unsigned char>(text[1])); }Answer
1 195
The substring is one byte of the two-byte letter, and that byte is 195.
-
Why can a byte count differ from a character count?
Answer
A character may need several bytes
UTF-8 uses one to four bytes for a code point.
The code on this page was built with g++ 15.2.0 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev