Data.Text

Text is not String

String is a list of Char: simple, lazy, and expensive, because every character is a cons cell. Data.Text is a packed array of UTF-16 code units with a length and an offset, so a character count is a subtraction of two integers instead of a traversal:

pack :: String -> Text unpack :: Text -> String length :: Text -> Int splitOn :: Text -> Text -> [Text]

The catch is that Text is strict in its content and has its own module of functions: map, filter and the list functions do not work on it untouched. The two conversions are pack and unpack, and a program that keeps Text at its edges and converts once is the usual shape.

pack, unpack and the functions

module Main (main) where import qualified Data.Text as T main :: IO () main = do let text = T.pack "Hello, Haskell world" print (T.length text) print (T.toUpper text) print (T.splitOn (T.pack ",") text) print (T.words text) print (T.replace (T.pack "Haskell") (T.pack "GHC") text) print (T.isPrefixOf (T.pack "Hello") text)

ghc -Wall -O1 -o text text.hs
./text

20
"HELLO, HASKELL WORLD"
["Hello"," Haskell world"]
["Hello,","Haskell","world"]
"Hello, GHC world"
True

Every function takes and returns Text, so each literal is wrapped once with pack and the printed form is the one of Text, with the quotes of the Show instance. The length is 20 characters, not 20 bytes and not 20 cons cells, and the split kept the space after the comma, which is why the second piece starts with one.

Trimming and splitting fields

A line of input usually needs trimming before it is compared, and empty fields are usually dropped. Both are one function each:

module Main (main) where import qualified Data.Text as T type Field = T.Text trim :: Field -> Field trim = T.strip clean :: [Field] -> [Field] clean = filter (not . T.null) . map trim main :: IO () main = do let raw = map T.pack [" ada ", "", " grace "] print (clean raw) print (T.intercalate (T.pack ";") (clean raw)) print (T.length (T.concat (clean raw))) print (T.unpack (T.take 2 (T.pack "abcdef")))

ghc -Wall -O1 -o fields fields.hs
./fields

["ada","grace"]
"ada;grace"
8
"ab"

The composition reads right to left: every field is trimmed, then the empty ones are dropped, and only then is the list printed. The length of the two joined names is 8 characters, so the concatenation happened after the trimming and the spaces are gone. T.take counts characters and not code points beyond the basic plane, which is the usual caveat of UTF-16 and worth knowing before a substring of an emoji.

String, Text or ByteString

A file read with readFile gives a String and a file read with Data.Text.IO.readFile gives a Text; both are in the boot libraries. The next lesson takes the byte-level view with ByteString, and the capstone reads a real file with the type that fits.

Exercises

  1. Count the words of a Text with T.words and compare with the String version.
  2. Split a line on ; and trim every field.
  3. Convert a Text to a String, upper-case it and convert back.
  4. Explain why T.length is not the number of bytes of the text on disk.
  5. Replace every occurrence of a word with a marker and print the result.
  6. Decide which type a program that reads a log file and counts its lines should use and why.

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. What is String in Haskell?

    Answer

    A list of Char

    String is [Char], so every character is a cons cell, which is why Text exists.

  2. How many characters are in T.pack "abc"?

    Answer

    3

    T.length counts characters, and abc has three.

  3. What does T.splitOn (T.pack ",") return?

    Answer

    A list of Text pieces around every comma

    splitOn cuts the text at each occurrence of the delimiter and keeps the pieces.

  4. What does this program print? module Main (main) where import qualified Data.Text as T main :: IO () main = print (T.length (T.pack "hello"))

    Answer

    5

    length gives an Int, so print shows 5 without quotes.

  5. Why does map toUpper not work on a Text directly?

    Answer

    Text is not a list, so the list function needs T.map

    Text has its own map, and the list version needs unpack and pack around it.

  6. What does this program print? module Main (main) where import qualified Data.Text as T main :: IO () main = print (T.words (T.pack "a b c"))

    Answer

    ["a","b","c"]

    T.words splits on runs of whitespace and drops the empty pieces.

  7. What does T.strip remove?

    Answer

    Whitespace from both ends

    strip trims the ends, while T.filter or T.replace would touch the inside.

  8. Which type is best for a large text file read once?

    Answer

    Text

    Text is packed and strict and has the functions, while a String pays a cons cell per character.

  9. What does T.unpack return?

    Answer

    A String

    unpack converts Text back to the list-of-Char representation.

  10. What does T.intercalate sep pieces do?

    Answer

    Joins the pieces with sep between them

    It is the join of a list of Text values, as the field program showed with a semicolon.

The code on this page was built with GHC 9.10.3 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev