Capstone: a CSV file
The I/O part ends with the file format every office program writes: comma-separated values. The simple version splits on the comma, and it is wrong as soon as a field contains one. The real format has a rule for that, a field may be wrapped in double quotes, and a quote inside such a field is written twice. This capstone writes the reader and the writer and then proves they agree.
The format and the data
The data is four rows of three fields: a header, a note with a comma inside, a city with a space and a note with quotes, and a city with a comma. The text in the program is written with those rules already applied, so the reader has something real to read:
name,city,note is the header. Ada,London,"likes tea, strong" is a row whose third field is quoted because it contains a comma. Grace,"New York","said ""hello"" once" has a quoted city (not strictly necessary) and a note with the two quotes of an escaped quote. Alan,"Cam, UK",plain is the same rule as the first case.
The reader
The reader walks the characters of one line and keeps three things: the fields it has finished, the field it is building and whether it is inside a quoted field. That is the whole state machine of the format:
import java.util.ArrayList;
import java.util.List;
public class Csv {
public static void main(String[] args) {
String text = String.join("\n",
"name,city,note",
"Ada,London,\"likes tea, strong\"",
"Grace,\"New York\",\"said \"\"hello\"\" once\"",
"Alan,\"Cam, UK\",plain");
List<List<String>> rows = parse(text);
for (List<String> row : rows) {
System.out.println(row.size() + " [" + String.join(" | ", row) + "]");
}
StringBuilder out = new StringBuilder();
for (List<String> row : rows) {
out.append(writeRow(row)).append('\n');
}
String written = out.toString();
System.out.print(written);
System.out.println("round trip: " + parse(written).equals(rows));
System.out.println("rows: " + rows.size());
}
static List<List<String>> parse(String text) {
List<List<String>> rows = new ArrayList<>();
for (String line : text.split("\n")) {
rows.add(parseRow(line));
}
return rows;
}
static List<String> parseRow(String line) {
List<String> fields = new ArrayList<>();
StringBuilder field = new StringBuilder();
boolean quoted = false;
for (int i = 0; i < line.length(); i++) {
char c = line.charAt(i);
if (quoted) {
if (c == '"') {
if (i + 1 < line.length() && line.charAt(i + 1) == '"') {
field.append('"');
i++;
} else {
quoted = false;
}
} else {
field.append(c);
}
} else if (c == '"') {
quoted = true;
} else if (c == ',') {
fields.add(field.toString());
field.setLength(0);
} else {
field.append(c);
}
}
fields.add(field.toString());
return fields;
}
static String writeRow(List<String> row) {
StringBuilder out = new StringBuilder();
for (int i = 0; i < row.size(); i++) {
if (i > 0) {
out.append(',');
}
String field = row.get(i);
if (field.contains(",") || field.contains("\"") || field.contains("\n")) {
out.append('"').append(field.replace("\"", "\"\"")).append('"');
} else {
out.append(field);
}
}
return out.toString();
}
}
javac -Xlint:all Csv.java
java Csv
3 [name | city | note]
3 [Ada | London | likes tea, strong]
3 [Grace | New York | said "hello" once]
3 [Alan | Cam, UK | plain]
name,city,note
Ada,London,"likes tea, strong"
Grace,New York,"said ""hello"" once"
Alan,"Cam, UK",plain
round trip: true
rows: 4
Quoted fields
The two rules are the whole difference from split. Inside a quoted field a comma is a character and not a separator, and two quotes in a row are one quote in the value. The reader above handles the first by ignoring the comma while quoted is true, and the second by consuming the pair and appending a single character.
The second row shows the result: the field is said "hello" once, with the quotes as text. A reader built on split(",") would have cut that field into three and the note of the first row into two, which is the bug this format exists to prevent.
The writer
The writer is the inverse and it quotes only when it must: a field with a comma, a quote or a newline goes inside quotes, and its own quotes are doubled. Every other field is written as it is, which keeps the file readable and keeps a simple file simple.
The condition is the same list in both directions, and that is not a coincidence: it is the definition of when the format needs the quotes. A writer that quoted everything would still be correct and would produce a file that a person cannot read; a writer that quoted nothing would produce a file that the reader splits wrongly.
The same reader on a file
The second program writes a small file, reads it back with Files.readAllLines and parses each line. It keeps the naive split on purpose, and the last line of its output is the number of fields that naive split finds in the row with a quoted comma:
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
public class CsvFile {
public static void main(String[] args) throws IOException {
Path file = Path.of("people.csv");
List<String> lines = List.of(
"name,city",
"Ada,London",
"Grace,\"New York\"");
Files.write(file, lines, StandardCharsets.UTF_8);
System.out.println("lines in file: " + Files.readAllLines(file, StandardCharsets.UTF_8).size());
List<List<String>> rows = new ArrayList<>();
for (String line : Files.readAllLines(file, StandardCharsets.UTF_8)) {
rows.add(splitSimple(line));
}
System.out.println("rows: " + rows.size() + " columns: " + rows.get(0).size());
System.out.println("second row: " + rows.get(1));
System.out.println("third row: " + rows.get(2));
Files.writeString(file, "Alan,\"Cam, UK\"\n", StandardCharsets.UTF_8, java.nio.file.StandardOpenOption.APPEND);
List<String> after = Files.readAllLines(file, StandardCharsets.UTF_8);
System.out.println("lines after append: " + after.size());
System.out.println("last line split on commas: " + splitSimple(after.get(3)).size());
Files.delete(file);
}
static List<String> splitSimple(String line) {
List<String> fields = new ArrayList<>();
for (String field : line.split(",")) {
fields.add(field.replace("\"", ""));
}
return fields;
}
}
javac -Xlint:all CsvFile.java
java CsvFile
lines in file: 3
rows: 3 columns: 2
second row: [Ada, London]
third row: [Grace, New York]
lines after append: 4
last line split on commas: 3
Three columns for a row that has two is the whole bug: the naive reader counts the comma inside Cam, UK as a separator. The first program, with its quote state, finds two. The file part is Files of lessons 105 and 106, and the append keeps the three lines that were already there.
The round trip
The last check is the one that proves the pair: the rows that were read are written and read again, and the two lists of rows must be equal. They are, which the true line says, and the number of rows is four.
The check is worth writing because the reader and the writer can be wrong in opposite directions and still pass a test that reads a file they also wrote. The round trip catches a reader that handles quotes and a writer that does not, and it is the reason the program prints the written text as well: a person can compare the two forms without running anything.
The production form of this program reads a file instead of a string, with Files.readAllLines of lesson 105, and writes with Files.write of lesson 106. The parser is the same code, because the format is the same whether the text came from a file, a socket or a String.
Exercises
- Add a field with a newline to the data and explain why the reader cannot handle it yet.
- Make the reader reject an unclosed quote with an exception that names the line.
- Write the rows to a file and read them back with the same two methods.
- Add a method that returns the column of a named header.
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
Why is
split(",")not enough to read CSV?Answer
A quoted field may contain a comma, which split treats as a separator
The quotes make a comma part of the value, and a character reader knows when they are open.
-
How is a double quote inside a quoted field written?
Answer
As two double quotes
The pair is the escape of the format, and the reader turns it back into one quote.
-
When does the writer put quotes around a field?
Answer
When it contains a comma, a quote or a newline
Those are the characters that would change the meaning of the field.
-
What state does the reader keep for one line?
Answer
The finished fields, the current field and whether a quote is open
Those three are enough to decide what a comma and a quote mean at each character.
-
Why does the writer quote only some fields?
Answer
Quotes are needed only where a character would change the meaning
A field without those characters reads the same without quotes, and the file stays readable.
-
Why is a round trip a stronger test than reading a file the program wrote?
Answer
It compares the values, so a reader and a writer wrong in opposite ways are caught
Two opposite bugs can agree on a text and disagree on the values, and the comparison sees that.
-
What does this program print?
public class Q1 { public static void main(String[] args) { String line = "Ada,London,tea"; String[] fields = line.split(","); System.out.println(fields.length); System.out.println(fields[1]); } }Answer
3 / London
A line without quotes splits into three fields, and the second is the city.
-
What does this program print?
public class Q2 { public static void main(String[] args) { String field = "say \"hi\""; String quoted = "\"" + field.replace("\"", "\"\"") + "\""; System.out.println(quoted); } }Answer
"say ""hi"""
The pair of quotes around the value plus the doubled quotes inside it.
-
Which method reads the lines of a CSV file into a list?
Answer
Files.readAllLines with a charset
The file method gives lines, and the parser of this lesson turns a line into fields.
-
What happens to a field that starts with a quote but never closes it?
Answer
The reader treats the rest of the line as part of the field
This parser has no check for it, which is the first exercise of the page.
The code on this page was built with OpenJDK 21.0.13 (javac and java) and run on Alpine Linux in a Docker container.
Article author: Arthur Isaev