Capstone B: the CSV core
The second capstone reads a CSV file, shows it in a grid and writes it back. The hard part is not the grid: it is the file. A CSV file has quoting rules, line endings, a byte order mark that arrives in the first column name, and rows whose column count disagrees with the header. This lesson writes the parser, the validator and the writer, and runs all three.
A parser, not a split
line.Split(',') is the parser that works on the test file and fails on the first address. The rule it misses is that a field may be quoted, and a quoted field may contain the separator, a doubled quote and a line break. The parser therefore walks one character at a time and keeps two pieces of state: the current field and whether it is inside quotes.
The states are few, and each one is a branch:
- Inside quotes, a doubled quote is one literal quote and the state stays quoted; a single quote ends the quoted field; every other character, including a comma and a newline, belongs to the field.
- Outside quotes, a quote starts a quoted field, a comma ends the field, a newline ends the field and the row, and a carriage return is skipped.
- At the end, a field or a row still in progress is flushed, so a file without a final newline loses nothing.
- A leading byte order mark is skipped, because a spreadsheet writes one and it would otherwise become part of the first header name.
The result is a header list and a list of rows, both of strings. The parser knows nothing about types; the validator is what gives a column a meaning.
Quotes and line breaks
The sample file of the run has three rows, and each one tests a rule. The first has a plain name; the second a quoted name with a comma; the third a quoted field with a real line break inside it. A viewer that loses the line break shows two rows and a wrong count, which is the bug this parser exists to avoid.
The difference between \r\n and \n is not a difference here: the carriage return is dropped outside quotes, and inside quotes it is kept, because a quoted field may legitimately contain one. The writer always writes \r\n, which is the line ending the format recommends and the one a spreadsheet expects.
Validation
The validator returns a list of messages, each with a one-based row number. A row with the wrong number of columns gets one message and no field checks, because indexing a short row would be the next bug. The fields are then checked in the order a reader would fix them: the id, the name and the total.
The total is parsed with decimal.TryParse, not double: money is decimal, and a total that passes through a binary floating point number can come back as 1250.0000000001. A negative total is a separate message from a non-numeric one, because they mean different mistakes.
Writing the table back
The writer quotes a field when it contains a comma, a quote or a line break, and doubles the quotes inside. A field without any of those is written bare, which keeps the file readable and the diff small. The round trip is the test: parse the written text again and compare it with the table, and write it a second time and compare the two texts.
The core in a program that runs
The parser, the validator and the writer in one program with a sample file and a deliberately bad one. It was built and run in the container:
using System;
using System.Collections.Generic;
using System.Text;
Console.WriteLine("a CSV reader with quoting and a validator");
var text = "id,name,total\r\nA-1,\"Lovelace, Ada\",1250.00\r\nA-2,\"Hopper, Grace\",340.50\r\nA-3,\"A note with a\nline break\",10.00\r\n";
var table = CsvTable.Parse(text);
Console.WriteLine($"rows: {table.Rows.Count}, columns: {table.Header.Count}");
foreach (var row in table.Rows)
{
Console.WriteLine($"{row[0]}: {row[1].Replace("\n", "\\n")} = {row[2]}");
}
var errors = table.Validate();
Console.WriteLine($"errors in the good file: {errors.Count}");
var bad = CsvTable.Parse("id,name,total\n,,\nB-1,Bob,-2\nB-2,Ann,abc\nB-3,Cid,5\nB-4,Dan\n");
foreach (var error in bad.Validate())
{
Console.WriteLine($"bad: {error}");
}
var written = table.ToCsv();
var again = CsvTable.Parse(written);
Console.WriteLine($"round trip: {again.Rows.Count == table.Rows.Count && written == table.ToCsv()}");
Console.WriteLine($"quoted comma kept: {written.Contains("\"Lovelace, Ada\"")}");
Console.WriteLine($"quoted newline kept: {written.Contains("\"A note with a\nline break\"")}");
internal sealed class CsvTable
{
private CsvTable(List<string> header, List<List<string>> rows)
{
Header = header;
Rows = rows;
}
public IReadOnlyList<string> Header { get; }
public IReadOnlyList<List<string>> Rows { get; }
public static CsvTable Parse(string text)
{
var rows = new List<List<string>>();
var row = new List<string>();
var field = new StringBuilder();
var quoted = false;
var index = 0;
if (text.Length > 0 && text[0] == '\uFEFF')
{
index = 1;
}
for (; index < text.Length; index++)
{
var c = text[index];
if (quoted)
{
if (c == '"' && index + 1 < text.Length && text[index + 1] == '"')
{
field.Append('"');
index++;
}
else if (c == '"')
{
quoted = false;
}
else
{
field.Append(c);
}
}
else if (c == '"')
{
quoted = true;
}
else if (c == ',')
{
row.Add(field.ToString());
field.Clear();
}
else if (c == '\n')
{
row.Add(field.ToString());
field.Clear();
rows.Add(row);
row = new List<string>();
}
else if (c != '\r')
{
field.Append(c);
}
}
if (field.Length > 0 || row.Count > 0)
{
row.Add(field.ToString());
rows.Add(row);
}
var header = rows.Count > 0 ? rows[0] : new List<string>();
var body = rows.Count > 1 ? rows.GetRange(1, rows.Count - 1) : new List<List<string>>();
return new CsvTable(header, body);
}
public List<string> Validate()
{
var errors = new List<string>();
for (var i = 0; i < Rows.Count; i++)
{
var row = Rows[i];
var number = i + 1;
if (row.Count != Header.Count)
{
errors.Add($"row {number}: expected {Header.Count} columns, found {row.Count}");
continue;
}
if (row[0].Trim().Length == 0)
{
errors.Add($"row {number}: id is empty");
}
if (row[1].Trim().Length == 0)
{
errors.Add($"row {number}: name is empty");
}
if (!decimal.TryParse(row[2], out var total))
{
errors.Add($"row {number}: total is not a number");
}
else if (total < 0m)
{
errors.Add($"row {number}: total must not be negative");
}
}
return errors;
}
public string ToCsv()
{
var builder = new StringBuilder();
AppendRow(builder, Header);
foreach (var row in Rows)
{
AppendRow(builder, row);
}
return builder.ToString();
}
private static void AppendRow(StringBuilder builder, IReadOnlyList<string> values)
{
for (var i = 0; i < values.Count; i++)
{
if (i > 0)
{
builder.Append(',');
}
builder.Append(Quote(values[i]));
}
builder.Append("\r\n");
}
private static string Quote(string value)
{
if (value.IndexOfAny(new[] { ',', '"', '\n', '\r' }) < 0)
{
return value;
}
return "\"" + value.Replace("\"", "\"\"") + "\"";
}
}
dotnet run
a CSV reader with quoting and a validator
rows: 3, columns: 3
A-1: Lovelace, Ada = 1250.00
A-2: Hopper, Grace = 340.50
A-3: A note with a\nline break = 10.00
errors in the good file: 0
bad: row 1: id is empty
bad: row 1: name is empty
bad: row 1: total is not a number
bad: row 2: total must not be negative
bad: row 3: total is not a number
bad: row 5: expected 3 columns, found 2
round trip: True
quoted comma kept: True
quoted newline kept: True
The output is the real run. The quoted comma stayed inside the field, the quoted newline stayed inside the field and is printed as \n so the line does not break the terminal output, the bad file produced six messages with the row numbers a user can act on, and the round trip came back identical. The last two lines are the writer's two quoting rules, tested against the text it produced.
Exercises
- Add a row with a doubled quote inside a quoted field and run the program; explain which branch handles it.
- Add a check that the header is exactly id,name,total and return one message when it is not.
- Write a file with a byte order mark in front of the header and check that the first column is still id.
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
Why is line.Split wrong for CSV?
Answer
A quoted field may contain the separator
The comma of an address is inside one quoted field.
-
What does a doubled quote inside a quoted field mean?
Answer
One literal quote
It is the CSV escape for a quote.
-
What happens to a newline inside a quoted field?
Answer
It is part of the field
The run keeps the embedded line break in the third row.
-
Why is a byte order mark skipped?
Answer
Otherwise it becomes part of the first column name
A spreadsheet writes one and the header must stay id.
-
What line ending does the writer produce?
Answer
CRLF
The writer appends \r\n after every row.
-
Why does a row with the wrong column count get no field messages?
Answer
Indexing a short row would throw
The run shows one message for that row.
-
Which type parses a total?
Answer
decimal
Money is decimal, not a binary float.
-
What did the bad file report for its second row?
Answer
total must not be negative
The run prints the six messages in row order.
-
When does the writer quote a field?
Answer
When it contains a comma, a quote or a line break
Bare fields keep the file readable.
-
What did the round trip print?
Answer
True, because the written text parses to the same table
Parse, write, compare is the test of the pair.
Every listing on this page is plain .NET code with no window in it: it was built and run with the .NET SDK 10.0.303 on Alpine Linux 3.24.3 (x86_64) in a Docker container, and the output printed below the program is that run. The WinForms pages of this course are built for net10.0-windows against Microsoft.WindowsDesktop.App.Ref 10.0.15 from the offline feed and never run on Linux: WinForms needs Windows.
Article author: Arthur Isaev