Golden files and diff

Some programs are easy to check by their output and hard to check any other way: a report, a formatter, a code generator, a compiler error message. There is no return value to compare and no function to call, only the text the program prints. The test for that is a file with the text it should print, and diff between the file and the run.

A golden file is the fixture, the output of a run somebody read and approved. The check is the same three steps every time: run the program into a file, compare the file with the fixture, and fail when they differ. This page does the three steps with a report program, a golden file and a shell script, and it shows both outcomes with the real diff output.

A test with a fixture

The comparison is byte for byte. A trailing space, a column that moved by one and a differently rounded number all fail, which is what makes the check useful for formatted output: diff shows the line that changed and the two versions of it.

The cost is the other side of the same coin. A golden file says what the output is, not what it should be, so a wrong output that somebody saved becomes the expected one. The rule that keeps it honest is the one in the chapter on updating a golden file: read the diff before you save it.

The program and its input

The program reads a file of comma-separated records, keeps the lines that parse, and prints a report: one line per region in alphabetical order and a total line. Each line is printf "%-10s %8d\n", so the amounts line up under each other and the alignment is part of the output the golden file holds.

module Main (main) where import Data.Char (isDigit) import Data.List (sortOn) import Data.Maybe (mapMaybe) import System.Environment (getArgs) import Text.Printf (printf) data Row = Row { rowName :: String , rowAmount :: Int } parseRow :: String -> Maybe Row parseRow line = case break (== ',') line of (name, ',':amount) | not (null name) , not (null amount) , all isDigit amount -> Just (Row name (read amount)) _ -> Nothing report :: [Row] -> String report rows = header ++ concatMap line (sortOn rowName rows) ++ footer where header = printf "%-10s %8s\n" "region" "total" footer = printf "%-10s %8d\n" "all" (sum (map rowAmount rows)) line row = printf "%-10s %8d\n" (rowName row) (rowAmount row) main :: IO () main = do args <- getArgs case args of [path] -> do content <- readFile path let rows = mapMaybe parseRow (lines content) putStr (report rows) _ -> putStrLn "usage: report FILE"

The build is clean under -Wall and hlint says No hints. mapMaybe parseRow is the filter-and-unwrap that hlint prefers over a list comprehension with a pattern in it, and sortOn rowName is the sort key that keeps the report stable.

The input is four lines, one of which has a region twice, because a report that silently merged them would be a report with a bug:

north,1200 south,700 north,150 east,95

The golden file and diff

The first run is the one that makes the fixture. The output is read once, line by line, and saved as golden.txt. From that moment the check is the diff, and its status is the status of the test.

ghc -Wall -O1 -o report Report.hs
./report records.txt > golden.txt

[1 of 2] Compiling Main             ( Report.hs, Report.o )
[2 of 2] Linking report

The file holds one line per region and a total. The amounts line up because of the two field widths, and the movement of a single space in the format string would show up in the diff:

region        total
east             95
north          1200
north           150
south           700
all            2145

The check runs the program into a file of its own and compares, so a failing run leaves both versions on disk. diff -u prints nothing and exits 0 when the files are equal, which is the pass:

./report records.txt > out.txt
diff -u golden.txt out.txt; echo "status $?"

status 0

A test whose only output is status 0 is still a test. The -u flag matters for the failing case, not for this one: it makes diff print three lines of context around each change and mark the old and new lines, which is what turns a failure into a readable report.

When the output changes

A fifth record is added to the input, a region the golden file has never seen. The program prints one more line and a larger total, and the diff names both changes with the lines around them:

north,1200 south,700 north,150 east,95 west,300

./report records2.txt > out.txt
diff -u golden.txt out.txt; echo "status $?"

--- golden.txt
+++ out.txt
@@ -3,4 +3,5 @@
 north          1200
 north           150
 south           700
-all            2145
+west            300
+all            2445
status 1

The two lines that start with - and + are the change, and the three lines above them are context so a reader can see where in the report it happened. A failing diff exits 1, which is the status the script below turns into the status of the test.

The check in a script

Five lines in a file, with set -eu so that the failing diff ends the script with its status. The input file is the argument, so the same script checks the second data file without an edit:

#!/bin/sh set -eu ./report "$1" > out.txt diff -u golden.txt out.txt echo "report matches the golden file"

sh run-golden.sh records.txt; echo "status $?"

report matches the golden file
status 0

The script prints its line only when the diff succeeded, because set -e stops it at the first command that fails. With the other data file the diff fails and the status is 1, and the message of a passing run never appears:

sh run-golden.sh records2.txt; echo "status $?"

--- golden.txt
+++ out.txt
@@ -3,4 +3,5 @@
 north          1200
 north           150
 south           700
-all            2145
+west            300
+all            2445
status 1

Updating a golden file

When the change is the one you wanted, the new output becomes the fixture. The order matters: look at the diff first, then save it. Saving first and looking afterwards is how a bug becomes the expected behaviour of a program.

sh run-golden.sh records2.txt
./report records2.txt > golden.txt
sh run-golden.sh records2.txt; echo "status $?"

--- golden.txt
+++ out.txt
@@ -3,4 +3,5 @@
 north          1200
 north           150
 south           700
-all            2145
+west            300
+all            2445
report matches the golden file
status 0

The three commands are one update: the expected failure, the copy that accepts it, and the run that now passes. A golden file is the only kind of test where the expected value is generated by the program under test, and that is why the diff comes before the copy in every real suite.

Where golden files hurt

A golden file does not localise a failure. The report above is seven lines and a reader can see the change; a compiler that prints four hundred lines gives a diff of the line that moved, and the line that moved is often a timestamp or a temporary path. Programs that print the time, a process id, an absolute path or a hash of their own binary need those fields masked before the comparison, and a mask is a second thing to get right.

Golden files also grow. A suite with two hundred of them has two hundred fixtures that nobody reads and one command that rewrites all of them, which is how a suite stops testing anything. Keep them for the outputs you would otherwise read by hand, keep them small, and read the diff of every update.

The next page measures which parts of a program a test run reached, with hpc. The formatting of the report itself is the subject of the course lesson on formatting output, and the streaming lesson of Capstone A reads the same kind of file without ever holding it in memory.

The code on this page was built with GHC 9.10.3 and run with the diff and sh of Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev