Golden files
Some programs are judged by their output as a whole: a report, a formatter, the diagnostics of a compiler. The test of such a program is a file that holds the expected text, and the comparison is a diff. The file is called golden because it is the answer the program must keep giving, and a diff is a better failure message than the words that the strings differ, because it shows the one line that moved.
The report
The program is the small report of the first capstone: three levels with a count each, and a total. The widths are in the format string, so the output is a table and not a list of numbers:
Console.WriteLine("report 1.0");
var rows = new[] { ("INFO", 3), ("WARN", 1), ("ERROR", 2) };
foreach (var (level, count) in rows)
{
Console.WriteLine($"{level,-6}{count,3}");
}
Console.WriteLine($"{"total",-6}{rows.Sum(row => row.Item2),3}");
A report has one property that makes it testable: it writes to standard output and nothing else, so the shell can put its output in a file without the program knowing. The banner line is part of the report on purpose, because a golden file that does not hold the banner would not notice a change to it.
The golden file
The golden file is the text below, saved next to the project. It was made by running the program once and keeping the output, which is how a golden file is born; from then on it is edited by hand and the test is what decides whether the program still matches it:
report 1.0
INFO 3
WARN 1
ERROR 2
total 6
The spaces matter. The level is padded to six characters and the count is right-aligned in three, so a golden file that has the same words in other columns is a different file. That is a feature: the output of a formatter is exactly this kind of text, and a test of it must compare the spaces.
A comparison
Three commands make the comparison: the run writes stdout to report.txt, diff compares the two files, and the shell prints the status of the last one. The order matters, because echo resets $? for everything after it:
dotnet run > report.txt
diff golden.txt report.txt
echo "status: $?"
status: 0
A diff of two identical files prints nothing and exits zero, so the status is the whole result. A script that runs a suite of golden files therefore prints the diff of the failure and stops on the first non-zero status, which is the shape of the script on the next page.
A change
The second comparison breaks the golden file on purpose: one count is three instead of two. The diff names the line and shows both values:
sed "s/ERROR 2/ERROR 3/" golden.txt > broken.txt
diff broken.txt report.txt
echo "status: $?"
--- broken.txt
+++ report.txt
@@ -1,5 +1,5 @@
report 1.0
INFO 3
WARN 1
-ERROR 3
+ERROR 2
total 6
status: 1
A line that begins with a minus is in the first file and a line that begins with a plus is in the second, which is why the command puts the golden file first: a reader reads "the golden file says three, the program printed two". The status is one, so the comparison is a test with a result and not a warning.
- A golden file that is updated by the test itself is not a test: the comparison must be against a file a person reviewed. A flag such as --write that rewrites the file is the usual way, and the diff of the rewritten file is what a reviewer reads.
- The text of a golden file depends on the culture and on the system's line ending, which is why a report that is compared this way formats with the invariant culture (lesson 075) and writes \n itself.
- A diff is a program of the system and not of .NET: on Linux it is in the image, and on Windows the same idea is fc or a comparison inside the test program.
The next page turns the status of these comparisons into the status of a suite: a driver, a shell script and the four measured statuses of the pair.
Exercises
- Add a fourth level to the report and update the golden file by hand.
- Compare with diff -u and describe how the output differs from the one above.
- Run the comparison from a script that stops at the first failure and prints the number of the file that failed.
- Write a second golden file for a run with a different culture and explain which one the test should use.
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
What is a golden file?
Answer
A file that holds the expected output of the program
The page keeps the report text in a file and compares it.
-
What is the status of diff when the two files are identical?
Answer
0
The measured comparison of the same file prints only status 0.
-
What does this program print?
Console.WriteLine($"{"INFO",-6}{3,3}");Answer
INFO 3
The name is padded to six characters and the count to three on the right.
-
What does a diff line that begins with a minus mean?
Answer
The line is in the first file and not the second
The command puts the golden file first, so a minus is the golden value.
-
What does this program print?
Console.WriteLine("a" + " " + "b");Answer
a b
One space between the two parts, because the middle string is one space.
-
Why is a golden file not rewritten by the test itself?
Answer
Then the test agrees with whatever the program prints
The comparison must be against a file a person reviewed.
-
What does this program print?
var text = "abc"; Console.WriteLine(text.Length);Answer
3
Three characters, which is what a spaces comparison also counts.
-
Why must a suite keep the status of diff before printing anything else?
Answer
The next command resets $?, so the status would be lost
The page says the status is read before echo appears.
-
What does this program print?
Console.WriteLine($"{42,4}|");Answer
42|
A positive width right-aligns the number in the column.
-
Which kind of program is tested best by a golden file?
Answer
One whose whole output is the answer, such as a report or a formatter
A diff of the whole output is the point of the file.
The code on this page was built with the .NET SDK 10.0.303 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev