Capstone A: a large file
The report of the previous lessons held five entries in a list. A log of a day does not fit in a list, so the reader takes one line at a time and keeps only what the report needs. This lesson writes two hundred thousand lines and reads them back with a live set that does not grow with the file.
Two hundred thousand lines
The program writes its own input, which is the only way a lesson can have a large file and still run the same on every machine. A StreamWriter buffers the text, so two hundred thousand lines are not two hundred thousand writes. The listing is the whole program, write and read together:
var path = Path.Combine(Path.GetTempPath(), "aredel-large.log");
const int Count = 200000;
using (var writer = new StreamWriter(path, false))
{
for (var i = 0; i < Count; i++)
{
writer.WriteLine($"2024-03-10T09:30:{i % 60:00}Z INFO request id={i} ms={i % 100}");
}
}
var before = GC.GetTotalMemory(true);
var lines = 0;
string? first = null;
string? last = null;
using (var reader = new StreamReader(path))
{
while (reader.ReadLine() is { } line)
{
first ??= line;
last = line;
lines++;
}
}
var after = GC.GetTotalMemory(false);
Console.WriteLine($"lines {lines}");
Console.WriteLine($"first {first}");
Console.WriteLine($"last {last}");
Console.WriteLine($"grew under 10 MB {after - before < 10 * 1024 * 1024}");
File.Delete(path);
dotnet run
lines 200000
first 2024-03-10T09:30:00Z INFO request id=0 ms=0
last 2024-03-10T09:30:19Z INFO request id=199999 ms=99
grew under 10 MB True
The live set
The last line is the measurement that matters: GC.GetTotalMemory(true) collected before the read, and GC.GetTotalMemory(false) asked again after it without collecting. The difference is the live set of the reader, which stays far below ten megabytes for a file of several megabytes, because a line is dropped as soon as the next one is read. A program that used File.ReadAllLines would hold every line at once and its live set would grow with the file.
The boolean, not the byte count, is on the page because the exact live set moves between runs with the buffer of the reader and the state of the runtime. What does not move is the shape: a streaming read is constant in the size of the file, and an eager read is linear.
Closing the reader
- using disposes the reader, which flushes nothing (a reader has nothing to flush) and closes the file handle.
- A reader that is never disposed keeps the file open until the garbage collector finalises it; on Linux a second process can still read the file, and on a busy server the handles run out.
- StreamReader detects the byte order mark of a file and assumes UTF-8 without one; the encoding can be named in its constructor.
- A line that is longer than the buffer is still returned whole; the reader grows its buffer rather than splitting the line.
The capstone continues in the next part with a command-line front end and a JSON form of the same report, and the last lesson of this part is a table of good and bad lines with a driver and an exit status.
Exercises
- Count only the ERROR lines of the large file without storing the others.
- Read the same file with File.ReadAllLines and compare the live set.
- Write the file with a UTF-8 byte order mark and see whether the first line changes.
- Break the read with break after a thousand lines and check that the file handle is closed.
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
Why does a large file use StreamReader instead of ReadAllLines?
Answer
It reads one line at a time, so the live set does not grow with the file
The measured read stayed under ten megabytes for two hundred thousand lines.
-
What does StreamReader.ReadLine return at the end of the file?
Answer
null
The while loop ends when the pattern sees null.
-
What does this program print?
using var reader = new StringReader("a\nb\n"); Console.WriteLine(reader.ReadLine());Answer
a
The first call returns the first line.
-
What does the pattern is { } line check?
Answer
That the value is not null, and names it
It is a null check and an assignment in one expression.
-
What does this program print?
using var reader = new StringReader("x"); Console.WriteLine(reader.ReadLine() is null);Answer
False
The reader returns the line x, so the value is not null.
-
What does using do to a StreamReader?
Answer
Disposes it at the end of the block, closing the file
The handle is released on every path out of the block.
-
What does this program print?
Console.WriteLine(199999 % 60);Answer
19
199999 divided by 60 leaves 19, which is the second of the last line.
-
Why compare GC.GetTotalMemory before and after the read?
Answer
To see whether the live set grew with the file
A streaming read keeps the live set small, which the boolean states.
-
What does this program print?
Console.WriteLine("a\nb".Split('\n').Length);Answer
2
One line break produces two parts.
-
Why does the program write the file before it reads it?
Answer
The lesson needs a large input that is the same on every machine
The program is self-contained and deletes the file at the end.
The code on this page was built with the .NET SDK 10.0.303 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev