Capstone A: a large file

The report of the previous lessons held five entries in a list. A log of a day does not fit in a list, so the reader takes one line at a time and keeps only what the report needs. This lesson writes two hundred thousand lines and reads them back with a live set that does not grow with the file.

Two hundred thousand lines

The program writes its own input, which is the only way a lesson can have a large file and still run the same on every machine. A StreamWriter buffers the text, so two hundred thousand lines are not two hundred thousand writes. The listing is the whole program, write and read together:

var path = Path.Combine(Path.GetTempPath(), "aredel-large.log"); const int Count = 200000; using (var writer = new StreamWriter(path, false)) { for (var i = 0; i < Count; i++) { writer.WriteLine($"2024-03-10T09:30:{i % 60:00}Z INFO request id={i} ms={i % 100}"); } } var before = GC.GetTotalMemory(true); var lines = 0; string? first = null; string? last = null; using (var reader = new StreamReader(path)) { while (reader.ReadLine() is { } line) { first ??= line; last = line; lines++; } } var after = GC.GetTotalMemory(false); Console.WriteLine($"lines {lines}"); Console.WriteLine($"first {first}"); Console.WriteLine($"last {last}"); Console.WriteLine($"grew under 10 MB {after - before < 10 * 1024 * 1024}"); File.Delete(path);

dotnet run

lines 200000
first 2024-03-10T09:30:00Z INFO request id=0 ms=0
last 2024-03-10T09:30:19Z INFO request id=199999 ms=99
grew under 10 MB True

The live set

The last line is the measurement that matters: GC.GetTotalMemory(true) collected before the read, and GC.GetTotalMemory(false) asked again after it without collecting. The difference is the live set of the reader, which stays far below ten megabytes for a file of several megabytes, because a line is dropped as soon as the next one is read. A program that used File.ReadAllLines would hold every line at once and its live set would grow with the file.

The boolean, not the byte count, is on the page because the exact live set moves between runs with the buffer of the reader and the state of the runtime. What does not move is the shape: a streaming read is constant in the size of the file, and an eager read is linear.

Closing the reader

The capstone continues in the next part with a command-line front end and a JSON form of the same report, and the last lesson of this part is a table of good and bad lines with a driver and an exit status.

Exercises

  1. Count only the ERROR lines of the large file without storing the others.
  2. Read the same file with File.ReadAllLines and compare the live set.
  3. Write the file with a UTF-8 byte order mark and see whether the first line changes.
  4. Break the read with break after a thousand lines and check that the file handle is closed.

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. Why does a large file use StreamReader instead of ReadAllLines?

    Answer

    It reads one line at a time, so the live set does not grow with the file

    The measured read stayed under ten megabytes for two hundred thousand lines.

  2. What does StreamReader.ReadLine return at the end of the file?

    Answer

    null

    The while loop ends when the pattern sees null.

  3. What does this program print? using var reader = new StringReader("a\nb\n"); Console.WriteLine(reader.ReadLine());

    Answer

    a

    The first call returns the first line.

  4. What does the pattern is { } line check?

    Answer

    That the value is not null, and names it

    It is a null check and an assignment in one expression.

  5. What does this program print? using var reader = new StringReader("x"); Console.WriteLine(reader.ReadLine() is null);

    Answer

    False

    The reader returns the line x, so the value is not null.

  6. What does using do to a StreamReader?

    Answer

    Disposes it at the end of the block, closing the file

    The handle is released on every path out of the block.

  7. What does this program print? Console.WriteLine(199999 % 60);

    Answer

    19

    199999 divided by 60 leaves 19, which is the second of the last line.

  8. Why compare GC.GetTotalMemory before and after the read?

    Answer

    To see whether the live set grew with the file

    A streaming read keeps the live set small, which the boolean states.

  9. What does this program print? Console.WriteLine("a\nb".Split('\n').Length);

    Answer

    2

    One line break produces two parts.

  10. Why does the program write the file before it reads it?

    Answer

    The lesson needs a large input that is the same on every machine

    The program is self-contained and deletes the file at the end.

The code on this page was built with the .NET SDK 10.0.303 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev

Banner image