Streams and encoding

A stream is a sequence of bytes a program reads or writes a piece at a time, and a file is one source of such a sequence. The File class of the previous lesson is a shortcut that opens a stream, moves the whole contents and closes it again; this lesson uses the stream itself, which is what a program needs for a file that does not fit in memory or that is still being written.

A FileStream reads bytes

FileStream works in bytes: no text, no lines, no encoding. The program writes one byte and then two more, opens the file again and reads what is there:

var path = Path.Combine(Path.GetTempPath(), "aredel-stream.bin"); using (var stream = new FileStream(path, FileMode.Create, FileAccess.Write)) { stream.WriteByte(65); stream.Write(new byte[] { 66, 67 }, 0, 2); } using (var stream = new FileStream(path, FileMode.Open, FileAccess.Read)) { var buffer = new byte[8]; var read = stream.Read(buffer, 0, buffer.Length); Console.WriteLine($"read {read}"); Console.WriteLine($"text {System.Text.Encoding.ASCII.GetString(buffer, 0, read)}"); Console.WriteLine($"length {stream.Length}"); Console.WriteLine($"position {stream.Position}"); } Console.WriteLine($"file {new FileInfo(path).Length}"); File.Delete(path);

dotnet run

read 3
text ABC
length 3
position 3
file 3

Read returned three, not eight: it reads what is there and stops at the end of the file. The position is the offset of the next byte, so it is 3 after the read, and the length of the file is the same number. A stream is not a file: the same code reads a network connection or a compressed block, which is why the whole API is built on this shape.

The StreamWriter buffer

A StreamWriter turns strings into bytes and keeps them in a buffer so that a small write does not become a system call. While the writer is open, the file can still be empty:

var path = Path.Combine(Path.GetTempPath(), "aredel-writer.txt"); using (var writer = new StreamWriter(path, false, new System.Text.UTF8Encoding(false))) { writer.WriteLine("alpha"); writer.WriteLine("beta"); Console.WriteLine($"inside using {new FileInfo(path).Length}"); } Console.WriteLine($"after using {new FileInfo(path).Length}"); using (var reader = new StreamReader(path)) { Console.WriteLine($"first {reader.ReadLine()}"); Console.WriteLine($"second {reader.ReadLine()}"); Console.WriteLine($"end {reader.ReadLine() is null}"); } var bytes = File.ReadAllBytes(path); Console.WriteLine($"bytes {bytes.Length}"); Console.WriteLine($"last {bytes[^1]}"); Console.WriteLine($"bom {bytes[0] == 0xEF}"); File.Delete(path);

dotnet run

inside using 0
after using 11
first alpha
second beta
end True
bytes 11
last 10
bom False

Inside the block the file is still 0 bytes, because the two lines sit in the buffer; leaving the block disposes the writer, which flushes the buffer and closes the handle, and the file grows to eleven bytes: six for alpha, five for beta, and one line break. The last byte is 10, the newline, and the first byte is not 0xEF, so no byte order mark was written. The third ReadLine returns null at the end of the file.

Flush pushes the buffer without closing the stream. A program that must make the data visible, a log that another process reads, calls it after each line; a program that writes and exits lets using do it.

Encoding and the byte order mark

The same four letters are five bytes in UTF-8 and four in Latin1. The program writes the same text in both encodings and reads the bytes:

using System.Text; var path = Path.Combine(Path.GetTempPath(), "aredel-encoding.txt"); File.WriteAllText(path, "café", new UTF8Encoding(false)); var bytes = File.ReadAllBytes(path); Console.WriteLine($"bytes {bytes.Length}"); Console.WriteLine($"last {bytes[^1]}"); Console.WriteLine($"ascii {bytes[0]}"); Console.WriteLine($"text {File.ReadAllText(path)}"); Console.WriteLine($"encoding {Encoding.UTF8.WebName}"); Console.WriteLine($"bom bytes {Encoding.UTF8.GetPreamble().Length}"); File.WriteAllText(path, "café", Encoding.Latin1); Console.WriteLine($"latin bytes {new FileInfo(path).Length}"); File.Delete(path);

dotnet run

bytes 5
last 169
ascii 99
text café
encoding utf-8
bom bytes 3
latin bytes 4

In UTF-8 the é takes two bytes, the last of them 169, and the three ASCII letters take one each. Latin1 has a single byte for each of the four characters, which is why the same text is one byte shorter. Encoding.UTF8.GetPreamble() returns the three-byte mark, but StreamWriter and File.WriteAllText write it only when the encoding object asks for it, and the default here does not.

Reading is the other direction: a byte sequence that is not valid UTF-8 becomes the replacement character instead of an exception, unless the decoder is created with a throwing fallback. A file from another system may be Latin1 or UTF-16, and a program that knows the encoding must say so; the default is UTF-8 and it is right for files this course writes.

Closing a stream

Every stream is an IDisposable, so a using block closes it on every path out, including an exception. A stream that is never closed keeps a file handle open until the garbage collector finalises it, which the lesson on finalizers measured; a process with many such streams runs out of handles before it runs out of memory. The next lesson keeps the same shape and puts a record through System.Text.Json.

Exercises

  1. Write ten bytes with FileStream and read them back one at a time with ReadByte; print each number.
  2. Call Flush after each line of the second program and see the length inside the block change.
  3. Write the same text in UTF-16 and compare the byte count with UTF-8.
  4. Open a file for reading and try to open it again for writing in the same program; read the exception if there is one.

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. What can a FileStream read and write?

    Answer

    Bytes

    The byte stream is the base; a reader or a writer adds the text layer.

  2. What does a StreamWriter do with the text before the buffer is full?

    Answer

    It keeps it in memory until Flush or Dispose

    The buffer is what made the file 0 bytes inside the using block.

  3. What does this program print? var path = Path.Combine(Path.GetTempPath(), "q.bin"); using (var s = new FileStream(path, FileMode.Create)) { s.WriteByte(1); s.WriteByte(2); s.WriteByte(3); } Console.WriteLine(new FileInfo(path).Length); File.Delete(path);

    Answer

    3

    Three bytes were written and the using block closed the stream.

  4. What does this program print? Console.WriteLine(System.Text.Encoding.UTF8.GetByteCount("café"));

    Answer

    5

    The accented letter takes two bytes in UTF-8, as the measured file did.

  5. Which method pushes the buffer to the file without closing the stream?

    Answer

    Flush

    Flush keeps the stream usable; Close and Dispose end it.

  6. What happens to a stream at the end of a using block?

    Answer

    Dispose is called, which closes it

    using is a try/finally that calls Dispose on every path.

  7. What does this program print? Console.WriteLine(System.Text.Encoding.ASCII.GetString(new byte[] { 65, 66, 67 }));

    Answer

    ABC

    Bytes 65, 66 and 67 are the letters A, B and C in ASCII.

  8. Which encoding does a StreamWriter use when the program passes none?

    Answer

    UTF-8 without a byte order mark

    That is the default, and the measured file started without 0xEF.

  9. What does FileMode.Create do when the file already exists?

    Answer

    Truncates it and starts from zero

    Create replaces the contents; Append is the mode that keeps them.

  10. What does this program print? Console.WriteLine(new System.Text.UTF8Encoding(false).GetPreamble().Length);

    Answer

    0

    The false argument asks for no byte order mark, so the preamble is empty.

The code on this page was built with the .NET SDK 10.0.303 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev