&str, UTF-8 and characters

In Rust &str is the borrowed form of text: a pointer to UTF-8 bytes and a length, with no capacity and no ownership. Every string literal is one, and every String can be borrowed as one. A function that only reads text should take &str, because it then accepts both.

Coming from C++? String views and slices writes the same first-word function with string_view and with &str, and then returns a view of a local in both languages: one compiles and reads freed memory, the other is stopped by the compiler.

The hard part of text is that bytes, characters and display width are three different counts. This lesson separates them: len counts bytes, chars().count() counts scalar values, and the byte index of a character comes from char_indices. Slicing uses byte indices, so it is the place where the three meet.

&str is the view

A parameter of type &str accepts a literal and a borrow of a String without a conversion. The function below takes the first character of its argument, and the two calls in main pass one of each.

The borrow ends with the call, so the owned string stays available. Nothing is copied into the function and nothing is allocated for the literal.

fn first_char(text: &str) -> char { text.chars().next().unwrap() } fn main() { let owned = String::from("rust"); println!("{} {}", first_char(&owned), first_char("borrowed")); }

rustc --edition 2024 str_params.rs
./str_params

r b

The two printed characters are the first of each word. The lesson on slices made the same point for &[T], and the rule is the same for text.

Bytes against characters

UTF-8 encodes a character in one to four bytes, and the accented letters of European languages take two. The program below prints both counts for a five-letter word with one accent: six bytes and five characters.

The difference matters for every buffer, column and limit in a user interface. A database column sized in bytes holds fewer accented characters than plain ones, and a terminal column is a third measure again, because some characters are wide.

fn main() { let text = "héllo"; println!("bytes: {}", text.len()); println!("chars: {}", text.chars().count()); }

rustc --edition 2024 byte_len.rs
./byte_len

bytes: 6
chars: 5

chars() walks the scalar values and allocates nothing, so counting is linear in the length of the text. The lesson on chars and bits shows the properties of a single character.

Byte indices of characters

char_indices yields each character with the byte index where its encoding starts. That is the only reliable way to turn a character position into a byte position, and it is what a highlighter or a parser uses to remember where a token began.

The program prints four lines for four characters: the indices jump from one to three because the accented letter occupies bytes one and two. The space and the final letter then continue from three and four.

fn main() { let text = "aé b"; for (index, ch) in text.char_indices() { println!("{index}: {ch}"); } }

rustc --edition 2024 char_indices.rs
./char_indices

0: a
1: é
3:
4: b

The iterator is lazy, so it stops as soon as the loop does. A parser that needs the position of a byte in the original text can keep the index and slice later, which is how the capstone tokenizer works.

Slicing on a boundary

A string slice takes byte indices, and they must fall on character boundaries. The program below splits a word into the first byte and the next two, which is one ASCII letter and the accented one, and then asks whether index one is a boundary.

The answer is true, because byte one starts the accented character. Slicing at byte two would panic: that position is in the middle of the encoding, and the result would not be valid UTF-8.

fn main() { let text = String::from("héllo"); let head = &text[..1]; let tail = &text[1..3]; println!("{head} {tail}"); println!("{}", text.is_char_boundary(1)); }

rustc --edition 2024 slicing.rs
./slicing

h é
true

is_char_boundary is the check to run before slicing when the index comes from outside. The lesson on parsing text uses it on input from a file.

A String is not a &str

A String owns its buffer and a &str borrows one, so assigning the first to a binding of the second type is E0308. The compiler suggests a borrow, which is free, and the fix is one ampersand.

The program below asks for a &str and builds a String. The error names both types and offers the borrow, which is the version that shares the buffer instead of copying it.

fn main() { let text: &str = String::from("hello"); println!("{text}"); }

rustc --edition 2024 str_type_err.rs

error[E0308]: mismatched types
--> str_type_err.rs:2:22
|
2 | let text: &str = String::from("hello");
| ---- ^^^^^^^^^^^^^^^^^^^^^ expected `&str`, found `String`
| |
| expected due to this
|
help: consider borrowing here
|
2 | let text: &str = &String::from("hello");
| +

error: aborting due to 1 previous error

For more information about this error, try `rustc --explain E0308`.

The reverse conversion, a &str into a String, needs an allocation and is written to_string() or String::from, which the previous lesson shows. The asymmetry is the point: borrowing is free, owning costs.

Exercises

  1. Write a function that returns the last character of a &str as a slice. Which byte index do you need, and how do you find it?
  2. Count the bytes and the characters of three words, one ASCII, one accented and one with a CJK character. Which difference is the largest?
  3. Slice a string at an index that is not a boundary and read the panic. Which position does the message name?

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. What is a &str?

    Answer

    A borrowed view of UTF-8 bytes

    A &str is a pointer and a length over text owned elsewhere.

  2. What does len() return for a &str?

    Answer

    The number of bytes

    The length is in bytes, which equals the character count only for ASCII.

  3. What does char_indices yield?

    Answer

    Each character with the byte index where it starts

    The pairs are character and byte position, which is what a slice needs.

  4. What happens when a string slice starts inside a character?

    Answer

    The program panics at run time

    The boundary is only known at run time, so the slice panics when it is not one.

  5. What does this program print? fn first_char(text: &str) -> char { text.chars().next().unwrap() } fn main() { let owned = String::from("rust"); println!("{} {}", first_char(&owned), first_char("borrowed")); }

    Answer

    r b

    Both arguments borrow as &str, and the first character of each word is printed.

  6. What does this program print? fn main() { let text = "héllo"; println!("bytes: {}", text.len()); println!("chars: {}", text.chars().count()); }

    Answer

    bytes: 6 / chars: 5

    The accented letter is two bytes and one character, so the counts differ by one.

  7. What does this program print? fn main() { let text = "aé b"; for (index, ch) in text.char_indices() { println!("{index}: {ch}"); } }

    Answer

    0: a / 1: é / 3: / 4: b

    The accented letter occupies bytes one and two, so the space starts at three.

  8. What does this program print? fn main() { let text = String::from("héllo"); let head = &text[..1]; let tail = &text[1..3]; println!("{head} {tail}"); }

    Answer

    h é

    The first byte is h and bytes one to two are the accented letter.

  9. What is the first line of the compiler output for this program? fn main() { let text: &str = String::from("hello"); println!("{text}"); }

    Answer

    error[E0308]: mismatched types

    The binding wants a borrowed view and the value is an owned String, and the help adds the ampersand.

  10. What does this program print? fn main() { let text = "hello"; println!("{}", text.is_char_boundary(2)); println!("{}", text.is_char_boundary(5)); }

    Answer

    true / true

    Every byte of ASCII text is a boundary, and the end of the string is one too.

The code on this page was built with rustc 1.96.1 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev

Banner image