Golden files and diff

A report program has no return value worth testing. Its result is the text it prints, and the test that suits text is a file that holds the text somebody approved: the golden file. The test runs the program, writes its output to a file of the same name, and asks diff whether the two are the same. The status of diff is the result.

The report under test

The program prints three region lines and a total, from a table in working storage, with edited amounts. It has no files and no arguments, which is what makes it a clean subject for a golden test: everything that decides its output is in the source.

IDENTIFICATION DIVISION. PROGRAM-ID. SALESREPORT. DATA DIVISION. WORKING-STORAGE SECTION. 01 T-AREA. 05 T-ROW OCCURS 3 TIMES. 10 T-REGION PIC X(2). 10 T-AMOUNT PIC 9(5)V99. 01 WS-I PIC 9. 01 WS-TOTAL PIC 9(7)V99 VALUE 0. 01 WS-AMT-OUT PIC Z(4)9.99. 01 WS-TOT-OUT PIC Z(6)9.99. PROCEDURE DIVISION. MOVE "N1" TO T-REGION(1) MOVE 100.00 TO T-AMOUNT(1) MOVE "S2" TO T-REGION(2) MOVE 250.50 TO T-AMOUNT(2) MOVE "E3" TO T-REGION(3) MOVE 75.25 TO T-AMOUNT(3) PERFORM VARYING WS-I FROM 1 BY 1 UNTIL WS-I > 3 MOVE T-AMOUNT(WS-I) TO WS-AMT-OUT DISPLAY "region " T-REGION(WS-I) " " FUNCTION TRIM(WS-AMT-OUT) ADD T-AMOUNT(WS-I) TO WS-TOTAL END-PERFORM MOVE WS-TOTAL TO WS-TOT-OUT DISPLAY "total " FUNCTION TRIM(WS-TOT-OUT) STOP RUN.

cobc -x -free -Wall -o report report.cob
./report

region N1 100.00
region S2 250.50
region E3 75.25
total 425.75

The golden file

The approved output, saved as golden.txt. It is a plain text file with the four lines and no trailing spaces, because a trailing space is the most common difference between two runs that look the same on a screen:

region N1 100.00 region S2 250.50 region E3 75.25 total 425.75

The file is in the repository beside the program, and it is reviewed like code: a change to it is a change to what the program promises.

The comparison that passes

./report > out.txt
diff -u golden.txt out.txt
echo $?

0

No output from diff and status 0: the two files are byte for byte the same. The redirect writes the file the comparison reads, so the test needs no temporary name and no cleanup, and a failing run leaves the actual output beside the golden one for whoever looks next.

In a script the three commands are one line, and the status of the line is the status of the test:

./report > out.txt && diff -u golden.txt out.txt

The comparison that fails

A one-cent change in one table entry, and the same comparison:

sed 's/MOVE 75.25 TO T-AMOUNT\(3\)/MOVE 75.26 TO T-AMOUNT\(3\)/' report.cob > report_bad.cob
cobc -x -free -Wall -o report_bad report_bad.cob
./report_bad > out.txt
diff -u golden.txt out.txt
echo $?

--- golden.txt
+++ out.txt
@@ -1,4 +1,4 @@
 region N1 100.00
 region S2 250.50
-region E3 75.25
-total 425.75
+region E3 75.26
+total 425.76
1

Two lines changed and two context lines, which is what a unified diff is for: the reader sees the old value, the new value and enough around them to know which line of a long report moved. The status is 1, so the suite fails and the build stops.

The diff in this image is the BusyBox one, and it prints the unified form even without -u; GNU diff prints the shorter 3,4c3,4 form unless it is given -u. Writing the flag in the script makes the output the same on both, which matters when the script runs on a build server and in a container.

Both forms carry the same information; the unified form wins because the context lines let a reader who does not have the file open still see where the change sits.

Keeping a golden file honest

The failure mode of golden tests is well known and worth writing down. When the diff shows up, the fastest way to a green build is to copy the new output over the golden file, and that blesses whatever changed, including a bug. The rule that keeps the file honest has three parts:

  1. Read the diff before the golden file is updated. Two lines of a report are cheap to read; a report of ten thousand lines is compared with diff and with wc -l, and only the changed lines are read.
  2. Say in the commit message why the output changed. "The fee is now 65 cents" is a reason; "updated the test" is not.
  3. Keep the amounts and the counts in the report itself, as the capstone report does with its grand total, so a diff of the totals is visible without arithmetic.

A golden file is also where line endings and encodings bite. The output of this program is ASCII with no trailing spaces, so the comparison is stable on any machine that runs it; a report with a date in it needs the date passed in, or the golden file fails every midnight for no reason. The shell runner of the previous page has the same shape with one line per case, and the next page asks which lines of the program the tests never reached.

The code on this page was built with cobc (GnuCOBOL) 3.2.0 and compared with the diff of Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev