Strings, Text and ByteString against std::string
std::string is a sequence of bytes with a length and a capacity, and so is a Haskell ByteString. String is a linked list of Unicode characters and Data.Text is a packed array of them. The text below is the same ten characters in all three, and it is twelve bytes in UTF-8, which is the first thing to know before comparing the types.
The two programs measure the text, upper-case it and split it on the space. Three of the four lines agree. The line that does not is the one about upper case, and it is the honest measure of what a byte container can do without a Unicode library.
std::string, bytes and a view
There is no character count in the standard library, so the program has one: a UTF-8 character starts at every byte that is not a continuation byte, which is the test (byte & 0xC0) != 0x80. Upper case is done by hand for the ASCII range only, because std::toupper works one byte at a time and in the C locale does nothing to a byte above 127: it cannot know that two of them form one letter.
#include <cstddef>
#include <cstdio>
#include <string>
#include <string_view>
std::size_t utf8_chars(std::string_view text) {
std::size_t count = 0;
for (unsigned char byte : text) {
if ((byte & 0xC0) != 0x80) {
++count;
}
}
return count;
}
std::string upper_ascii(std::string text) {
for (char& letter : text) {
if (letter >= 'a' && letter <= 'z') {
letter = static_cast<char>(letter - 'a' + 'A');
}
}
return text;
}
int main() {
const std::string text = "naïve café";
std::printf("bytes %zu\n", text.size());
std::printf("chars %zu\n", utf8_chars(text));
std::printf("upper %s\n", upper_ascii(text).c_str());
const std::size_t space = text.find(' ');
std::printf("split %s|%s\n", text.substr(0, space).c_str(), text.substr(space + 1).c_str());
}
substr takes byte positions, and the space between the two words is one byte, so the split works on this text. It would not work on a text whose separator is a multi-byte character, which is the point of the next chapter.
g++ -std=c++23 -Wall -Wextra -O2 strings.cpp -o strings
./strings; echo "status $?"
bytes 12
chars 10
upper NAïVE CAFé
split naïve|café
status 0
String, Text and ByteString
Three types for one value, and the program uses all three for what each is good at. Data.Text holds the characters; encoding it with encodeUtf8 gives a ByteString, whose length is the byte count; T.length is the character count. T.toUpper does the full Unicode case mapping, so ï becomes Ï. The first line of main sets the encoding of standard output to UTF-8, because a Haskell handle takes its encoding from the locale and a program under a C locale cannot print the accented letters at all; the last chapter shows that failure.
{-# LANGUAGE OverloadedStrings #-}
module Main (main) where
import qualified Data.ByteString as BS
import qualified Data.Text as T
import qualified Data.Text.Encoding as TE
import System.IO (hSetEncoding, stdout, utf8)
import Text.Printf (printf)
text :: T.Text
text = "naïve café"
main :: IO ()
main = do
hSetEncoding stdout utf8
let bytes = TE.encodeUtf8 text
printf "bytes %d\n" (BS.length bytes)
printf "chars %d\n" (T.length text)
printf "upper %s\n" (T.unpack (T.toUpper text))
let (left, rest) = T.breakOn " " text
printf "split %s|%s\n" (T.unpack left) (T.unpack (T.drop 1 rest))
T.breakOn splits on a Text rather than a byte, so a separator in any encoding works. The build is clean under -Wall and hlint says No hints:
ghc -Wall -O1 -o strings-hs Strings.hs
./strings-hs; echo "status $?"
[1 of 2] Compiling Main ( Strings.hs, Strings.o )
[2 of 2] Linking strings-hs
bytes 12
chars 10
upper NAÏVE CAFÉ
split naïve|café
status 0
An encoding is a layer
The four lines say what each type is. std::string and ByteString agree on 12 because both count bytes. Text and String agree on 10 because both count characters. The upper case line is the one that separates a byte container from a text type: without a Unicode library, C++ upper-cases the ASCII letters and leaves the two accented ones as they were, which is the correct answer for a language that treats a string as bytes and a wrong one for a language that treats it as text.
| Type | Holds | length counts | Cost of one character |
|---|---|---|---|
| std::string | bytes | bytes | one byte in UTF-8 |
| ByteString | bytes | bytes | one byte in UTF-8 |
| Text | characters, packed | characters | two bytes or four |
| String | characters, a list | characters | a cons cell and a boxed Char |
That last row is why String is the type to use for a line of input and not for a file. A list node is three words and a Char is one more, so a megabyte of text costs tens of megabytes as a String and one megabyte as a ByteString. The throughput page measures the difference on the same machine.
C++ has the same choice in a different shape. std::string is the default and it is already contiguous, so there is no slow string type to avoid; what it lacks is the character layer, and a program that needs one adds a library (ICU, or a UTF-8 wrapper of its own). std::string_view is the borrowed view that Haskell spells as a slice of a ByteString: no copy, valid while the owner lives, and the compiler does not track the lifetime in either language.
The column that moves
The quiet mistake is a field width. printf("%-10s") pads to ten bytes in C++, because that is what the conversion has to work with, and a name with an accented letter takes six of those bytes for five characters. The table below is the same in both languages; only one of them lines up:
#include <cstdio>
int main() {
std::printf("|%-10s|%4d|\n", "ada", 1);
std::printf("|%-10s|%4d|\n", "naïve", 2);
std::printf("|%-10s|%4d|\n", "linus", 3);
}
g++ -std=c++23 -Wall -Wextra -O2 Width.cpp -o width
./width; echo "status $?"
|ada | 1|
|naïve | 2|
|linus | 3|
status 0
The second row ends one column early, and nothing warns: the format is satisfied and the output is valid UTF-8, it is just not the table the programmer drew. The Haskell version pads by characters, because a String is a list of them and Text.Printf asks for its length:
module Main (main) where
import System.IO (hSetEncoding, stdout, utf8)
import Text.Printf (printf)
main :: IO ()
main = do
hSetEncoding stdout utf8
printf "|%-10s|%4d|\n" ("ada" :: String) (1 :: Int)
printf "|%-10s|%4d|\n" ("naïve" :: String) (2 :: Int)
printf "|%-10s|%4d|\n" ("linus" :: String) (3 :: Int)
ghc -Wall -O1 -o width-hs Width.hs
./width-hs; echo "status $?"
[1 of 2] Compiling Main ( Width.hs, Width.o )
[2 of 2] Linking width-hs
|ada | 1|
|naïve | 2|
|linus | 3|
status 0
The fix in C++ is the same character count the first program wrote, applied to the padding: count the code points, subtract from the width, and print the difference as spaces. That is a five-line helper and a decision to keep it; in Haskell the count is the type.
The handle that cannot encode
The C++ program above never worried about the locale, because a std::string is written byte for byte and the bytes are already UTF-8. A Haskell handle is a text device: it has an encoding, it takes it from the locale, and a locale of C means ASCII. The same table without the hSetEncoding line, under LANG=C:
module Main (main) where
import Text.Printf (printf)
main :: IO ()
main = do
printf "|%-10s|%4d|\n" ("ada" :: String) (1 :: Int)
printf "|%-10s|%4d|\n" ("naïve" :: String) (2 :: Int)
printf "|%-10s|%4d|\n" ("linus" :: String) (3 :: Int)
ghc -Wall -O1 -o nolocale-hs NoLocale.hs
LANG=C ./nolocale-hs; echo "status $?"
[1 of 2] Compiling Main ( NoLocale.hs, NoLocale.o )
[2 of 2] Linking nolocale-hs
|ada | 1|
|nanolocale-hs: <stdout>: commitBuffer: invalid argument (cannot encode character '\239')
HasCallStack backtrace:
collectBacktraces, called at libraries/ghc-internal/src/GHC/Internal/Exception.hs:169:13 in ghc-internal:GHC.Internal.Exception
toExceptionWithBacktrace, called at libraries/ghc-internal/src/GHC/Internal/IO.hs:260:11 in ghc-internal:GHC.Internal.IO
throwIO, called at libraries/ghc-internal/src/GHC/Internal/IO/Exception.hs:315:19 in ghc-internal:GHC.Internal.IO.Exception
ioException, called at libraries/ghc-internal/src/GHC/Internal/IO/Exception.hs:319:20 in ghc-internal:GHC.Internal.IO.Exception
status 1
The first row is on the screen and the second one stops in the middle: the handle had already written |na and then refused the byte for ï. The error is not about the string but about the device, and the fix is one line at the start of the program, or a locale that says UTF-8. This container runs with no locale set, which is why every program on this page that prints text sets the handle itself.
The Haskell lesson on the subject is Data.Text, and the next page in this set is STM against a mutex.
The C++ program was built with g++ 15.2.0 (Alpine 15.2.0) using -std=c++23 -Wall -Wextra -O2 and the Haskell program with GHC 9.10.3 using -Wall -O1, both run on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev