Text processing
Text arrives as bytes with separators: words, lines, key-value pairs. The C library has a function for each shape, but the important part is the ownership: some functions write into the buffer they receive, and some allocate a new one. This lesson uses both kinds on the same input.
Splitting with strtok
strtok replaces each delimiter with a zero and returns the start of the token. The first call receives the string and later calls pass NULL to continue, which means the function keeps state between calls. The delimiters are a set of characters, so "=;" splits on both.
#include <stdio.h>
#include <string.h>
int main(void)
{
char text[] = "name=Ada;role=admin;active=yes";
char *token = strtok(text, "=;");
while (token != NULL) {
printf("[%s]\n", token);
token = strtok(NULL, "=;");
}
return 0;
}
gcc -Wall -Wextra tokens.c -o tokens
./tokens
[name]
[Ada]
[role]
[admin]
[active]
[yes]
The text is a char array, not a string literal, because the function writes the zeros into it. The tokens alternate between keys and values in this input; a parser that needs the pairs calls the function twice per field or reads the original buffer after the split.
getline reads any line
fgets stops at the buffer size, so a long line arrives in pieces. getline from POSIX allocates a buffer and grows it until the whole line fits, returning its length with the newline included. The buffer must be freed once, and NULL with capacity zero starts it.
#define _POSIX_C_SOURCE 200809L
#include <stdio.h>
#include <stdlib.h>
int main(void)
{
char *line = NULL;
size_t capacity = 0;
ssize_t length;
while ((length = getline(&line, &capacity, stdin)) != -1) {
printf("length %zd: %s", length, line);
}
free(line);
return 0;
}
gcc -Wall -Wextra lines.c -o lines
printf 'first\nsecond line\n' | ./lines
length 6: first
length 12: second line
The length counts the newline, so the first line is five characters plus one. The %zd conversion is the correct one for ssize_t, which is a signed size_t.
Trimming a key and a value
strspn counts the characters from a set at the start, and strchr finds the first of a character. Together they trim the spaces of a small key-value line without altering the input outside the field. This is the parsing shape a configuration reader uses.
#include <stdio.h>
#include <string.h>
int main(void)
{
char line[] = " key = value ";
size_t start = strspn(line, " ");
size_t end = strlen(line);
while (end > start && line[end - 1] == ' ') {
end--;
}
line[end] = '\0';
char *equals = strchr(line + start, '=');
if (equals != NULL) {
*equals = '\0';
size_t key_end = strlen(line);
while (key_end > start && line[key_end - 1] == ' ') {
key_end--;
}
line[key_end] = '\0';
char *value = equals + 1;
while (*value == ' ') {
value++;
}
printf("key [%s]\n", line + start);
printf("value [%s]\n", value);
}
return 0;
}
gcc -Wall -Wextra fields.c -o fields
./fields
key [key]
value [value]
The pointers point into the same buffer, and the writes only add zeros, so no allocation is needed. The trailing spaces are removed before the split and the key is trimmed after it, which is why the two brackets contain exactly the words.
Counting words
A word count needs one flag, not an array of words: the flag says whether the previous character was inside a word, and a transition from outside to inside increments the count. The character class comes from ctype.h, the subject of lesson 24.
#include <ctype.h>
#include <stdio.h>
int main(void)
{
char text[] = "the quick brown fox";
int words = 0;
int in_word = 0;
for (int i = 0; text[i] != '\0'; i++) {
if (isspace((unsigned char) text[i])) {
in_word = 0;
} else if (!in_word) {
in_word = 1;
words++;
}
}
printf("words: %d\n", words);
return 0;
}
gcc -Wall -Wextra words.c -o words
./words
words: 4
Punctuation counts as part of a word under this rule, so "fox." would count the same as "fox". A stricter counter checks isalpha and starts a word only on a letter, which is the change the first exercise asks for.
strtok writes into its input
The first split replaced delimiters with zeros in the caller's buffer. A string literal lives in read-only memory, so passing one to strtok compiles and then crashes on the first write. No compiler warning appears, because the function parameter is an ordinary char *.
#include <stdio.h>
#include <string.h>
int main(void)
{
char *text = "a,b,c";
char *token = strtok(text, ",");
printf("%s\n", token);
return 0;
}
gcc -Wall -Wextra literal.c -o literal
./literal; echo "exit: $?"
exit: 139
The shell reports 139 for the segmentation fault. The fix is the array from the first section: char text[] = "a,b,c"; copies the bytes into writable storage, and the split works. The same rule applies to any function that takes a non-const char *.
Exercises
- Change the word count to count only sequences of letters, so "fox." is one word and "--" is none.
- Parse the line "a=1;b=2;c=3" into keys and values with strtok and print a table.
- Read a file with getline and print the longest line with its length.
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
What does
strtokdo to the string it receives?Answer
It writes zeros over the delimiters
The function replaces each delimiter with a terminator, which is why the input must be writable.
-
What does
getlinereport as its return value?Answer
The line length, including the newline
The signed length counts the bytes read, and the newline is part of the line when it is present.
-
What does
strspn(line, " ")return for a line that starts with three spaces?Answer
The number 3, the offset of the first non-space
It counts the leading characters that belong to the set, which is the offset of the first non-space.
-
How does the word counter avoid storing the words?
Answer
It tracks whether the previous byte was in a word
A transition from outside a word to inside one increments the count, so one flag is enough.
-
What does this program print?
#include <stdio.h> #include <string.h> int main(void){char s[]="name=Ada;role=admin;active=yes";for(char*t=strtok(s,"=;");t;t=strtok(NULL,"=;"))printf("[%s]",t);printf("\n");return 0;}Answer
[name][Ada][role][admin][active][yes]
The delimiter set contains both
=and;, so every key and value becomes a separate token. -
What does this program print for the two lines?
#define _POSIX_C_SOURCE 200809L #include <stdio.h> #include <stdlib.h> int main(void){FILE*f=tmpfile();fputs("first\nsecond line\n",f);rewind(f);char*l=NULL;size_t c=0;ssize_t n;while((n=getline(&l,&c,f))!=-1)printf("%zd ",n);printf("\n");free(l);fclose(f);return 0;}Answer
6 12
Each length includes the newline, so five and eleven characters become six and twelve.
-
What does this program print?
#include <stdio.h> #include <string.h> int main(void){char line[]=" key = value ";size_t start=strspn(line," "),end=strlen(line);while(end>start&&line[end-1]==' ')end--;line[end]=0;char*eq=strchr(line+start,'=');if(eq){*eq=0;size_t ke=strlen(line);while(ke>start&&line[ke-1]==' ')ke--;line[ke]=0;char*v=eq+1;while(*v==' ')v++;printf("key [%s]\nvalue [%s]\n",line+start,v);}return 0;}Answer
key [key] / value [value]
The leading and trailing spaces are removed around the key and the value before printing.
-
How many words does the counter report for
the quick brown fox?Answer
4
Each of the four runs of non-space characters starts a word, and the loop counts four transitions.
-
What happens when
strtokis given a string literal?Answer
It writes to read-only memory and crashes
The literal may live in read-only memory, and the first delimiter write ends the process with status 139.
-
What must happen to the buffer allocated by
getline?Answer
It is freed once after the loop
getlinegrows the buffer as needed, and onefreeat the end releases the final allocation.
The code on this page was run with gcc (Alpine 15.2.0) 15.2.0 on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev