Capstone C: the test harness
The module has an interface, a version and three status codes. What it does not have is a way to prove that it still behaves after a change, and in this image there is no COBOL test framework to lean on: no cobol-check, no zUnit, no GnuCOBOL test runner. A COBOL program that calls the module and compares the answers is the whole of the tooling, and it is enough.
A harness in COBOL
A driver program holds a table of cases: a request and the answer it must produce. For each case it moves the request into the copybook record, CALLs the module, compares the status and the result, and counts. At the end it prints one line with the two counts and returns 1 if anything failed, which is the number a build script and a person both read.
The comparison is exact on purpose. A test that checks "no crash" passes when the module returns zero for every input, and a test that checks the status and the exact number of cents fails when the rounding changes by one. In a money program the exact number is the requirement.
The harness
The file holds two programs. The first is the harness: five cases in a table, a loop that calls the module with each and compares the status and the cents, and a summary that sets the exit status. The second is the module, the same one as in the previous lesson:
IDENTIFICATION DIVISION.
PROGRAM-ID. HARNESS.
DATA DIVISION.
WORKING-STORAGE SECTION.
01 WS-CASES.
05 WS-CASE OCCURS 5 TIMES.
10 C-VERSION PIC 9(2).
10 C-OP PIC X(3).
10 C-CENTS PIC 9(9).
10 C-STATUS PIC 9(2).
10 C-EXPECT PIC S9(11).
01 WS-I PIC 9(2).
01 WS-PASSED PIC 9(3) VALUE 0.
01 WS-FAILED PIC 9(3) VALUE 0.
01 REQUEST-REC.
05 REQ-VERSION PIC 9(2).
05 REQ-OP PIC X(3).
05 REQ-CENTS PIC 9(9).
01 ANSWER-REC.
05 ANS-VERSION PIC 9(2).
05 ANS-STATUS PIC 9(2).
05 ANS-CENTS PIC S9(11).
05 ANS-MESSAGE PIC X(30).
PROCEDURE DIVISION.
MAIN-PARA.
PERFORM FILL-CASES
PERFORM VARYING WS-I FROM 1 BY 1 UNTIL WS-I > 5
PERFORM RUN-CASE
END-PERFORM
DISPLAY "passed " WS-PASSED " failed " WS-FAILED
IF WS-FAILED IS GREATER THAN 0
MOVE 1 TO RETURN-CODE
END-IF
STOP RUN.
FILL-CASES.
MOVE 1 TO C-VERSION(1)
MOVE "TAX" TO C-OP(1)
MOVE 10000 TO C-CENTS(1)
MOVE 0 TO C-STATUS(1)
MOVE 1300 TO C-EXPECT(1)
MOVE 1 TO C-VERSION(2)
MOVE "TAX" TO C-OP(2)
MOVE 25000 TO C-CENTS(2)
MOVE 0 TO C-STATUS(2)
MOVE 3250 TO C-EXPECT(2)
MOVE 1 TO C-VERSION(3)
MOVE "NET" TO C-OP(3)
MOVE 50000 TO C-CENTS(3)
MOVE 0 TO C-STATUS(3)
MOVE 43500 TO C-EXPECT(3)
MOVE 2 TO C-VERSION(4)
MOVE "TAX" TO C-OP(4)
MOVE 10000 TO C-CENTS(4)
MOVE 8 TO C-STATUS(4)
MOVE 0 TO C-EXPECT(4)
MOVE 1 TO C-VERSION(5)
MOVE "XXX" TO C-OP(5)
MOVE 10000 TO C-CENTS(5)
MOVE 9 TO C-STATUS(5)
MOVE 0 TO C-EXPECT(5).
RUN-CASE.
MOVE C-VERSION(WS-I) TO REQ-VERSION
MOVE C-OP(WS-I) TO REQ-OP
MOVE C-CENTS(WS-I) TO REQ-CENTS
CALL "TAXCALC" USING REQUEST-REC ANSWER-REC
IF ANS-STATUS NOT = C-STATUS(WS-I)
PERFORM FAIL-CASE
ELSE
IF ANS-CENTS NOT = C-EXPECT(WS-I)
PERFORM FAIL-CASE
ELSE
ADD 1 TO WS-PASSED
DISPLAY "pass case " WS-I
END-IF
END-IF.
FAIL-CASE.
ADD 1 TO WS-FAILED
DISPLAY "FAIL case " WS-I " status " ANS-STATUS
" cents " ANS-CENTS " expected " C-STATUS(WS-I)
" " C-EXPECT(WS-I).
END PROGRAM HARNESS.
IDENTIFICATION DIVISION.
PROGRAM-ID. TAXCALC.
DATA DIVISION.
WORKING-STORAGE SECTION.
01 WS-RATE PIC 9V99 VALUE 0.13.
LINKAGE SECTION.
01 REQUEST-REC.
05 REQ-VERSION PIC 9(2).
05 REQ-OP PIC X(3).
05 REQ-CENTS PIC 9(9).
01 ANSWER-REC.
05 ANS-VERSION PIC 9(2).
05 ANS-STATUS PIC 9(2).
05 ANS-CENTS PIC S9(11).
05 ANS-MESSAGE PIC X(30).
PROCEDURE DIVISION USING REQUEST-REC ANSWER-REC.
MOVE 1 TO ANS-VERSION
MOVE 0 TO ANS-CENTS
MOVE SPACES TO ANS-MESSAGE
IF REQ-VERSION NOT = 1
MOVE 8 TO ANS-STATUS
MOVE "unsupported request version" TO ANS-MESSAGE
GOBACK
END-IF
EVALUATE REQ-OP
WHEN "TAX"
COMPUTE ANS-CENTS ROUNDED = REQ-CENTS * WS-RATE
MOVE 0 TO ANS-STATUS
MOVE "tax calculated" TO ANS-MESSAGE
WHEN "NET"
COMPUTE ANS-CENTS = REQ-CENTS - 6500
MOVE 0 TO ANS-STATUS
MOVE "net after fee" TO ANS-MESSAGE
WHEN OTHER
MOVE 9 TO ANS-STATUS
MOVE "unknown operation" TO ANS-MESSAGE
END-EVALUATE
GOBACK.
END PROGRAM TAXCALC.
The five cases cover the three status codes and the rounding: a tax of a hundred, a tax of two hundred and fifty, a net of five hundred, a request with version 2 and a request with an unknown operation. The expected cents are the numbers the module produced on the previous page, written down as constants, so a change in either side shows up as a failure rather than as a new output somebody has to notice.
The run that passes
cobc -x -free -Wall -o harness harness.cob
./harness
echo $?
pass case 01
pass case 02
pass case 03
pass case 04
pass case 05
passed 005 failed 000
0
Five passes and status 0, which is what a build script wants to see. The harness is built from the same file as the module, so the compiler and the test agree about the interface by construction: a record that changed in one program and not the other fails the build before it fails a test.
The run that fails
A harness nobody has seen fail is a harness nobody trusts. Change one expected value, one digit, and run it again:
sed 's/MOVE 1300 TO C-EXPECT(1)/MOVE 1350 TO C-EXPECT(1)/' \
harness.cob > harness_bad.cob
sed -i 's/PROGRAM-ID. HARNESS./PROGRAM-ID. HARNESSB./; s/END PROGRAM HARNESS./END PROGRAM HARNESSB./' \
harness_bad.cob
cobc -x -free -Wall -o harness_bad harness_bad.cob
./harness_bad
echo $?
FAIL case 01 status 00 cents +00000001300 expected 00 +00000001350
pass case 02
pass case 03
pass case 04
pass case 05
passed 004 failed 001
1
Four passes, one failure, status 1, and the failing line carries the actual answer, the expected one and the case number. That is the whole diagnostic a person needs, and it is on one line because a harness that prints a paragraph per failure is a harness whose output nobody reads.
The second sed renames the program in both places the name appears, the PROGRAM-ID and the END PROGRAM; a COBOL file whose END PROGRAM names a different program than its PROGRAM-ID is a compile error. Renaming is worth the two lines: two main programs with one name in one folder confuse the runtime, which resolves a name to the first it finds.
A harness in a suite
The harness is a program with a status, so it composes with everything that already understands statuses. A shell line runs it and stops on a failure:
cobc -x -free -Wall -o harness harness.cob taxcalc.cob || exit 2
./harness || exit 1
echo "taxcalc: all cases passed"
Three properties make the pattern worth keeping. The exit status is the only interface a build server needs, so the harness can sit in a Makefile beside the production build. The cases are data in the program, so adding one is a five-line change and not a new framework. And the harness is plain COBOL, so the person who maintains the module can read and extend it on the day the module changes, which is not true of a test written in a language they do not use.
What the harness does not do is as important. It does not test the batch job of the previous lesson, the file handling or the parsing of the request line; those need a program that reads a file and compares its output, which is a golden-file test and a topic of its own. It does not measure coverage: this image has no COBOL coverage tool, and a number produced by a tool that does not exist here would be a number invented.
Capstone C ends here: a contract in a copybook, a module that implements it, a batch job that uses it and a harness that proves it. The three capstones together are the course in miniature: a file layout, an index and a report in A and B, an interface and its tests in C.
Exercises
- Add a case for an amount of one cent. What does the module answer, and is the rounding what you expected?
- Add a case with an empty operation. Which status does it get, and which side of the interface decides that?
- Make the harness print the failing request as well as the answer. Which fields do you add to the line?
- Change the module\'s rate to 0.20 and run the harness without touching the cases. How many fail, and what does that say about the table?
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
What does the harness return when a case fails?
Answer
1
The failed run printed passed 004 failed 001 and status 1.
-
Why compare the exact number of cents?
Answer
In a money program the exact result is the requirement, and a changed round must fail
The harness checks ANS-STATUS and ANS-CENTS exactly.
-
What does the failing line of the mutated run show?
Answer
The case number, the actual status and cents, and the expected ones
FAIL case 01 status 00 cents +00000001300 expected 00 +00000001350.
-
Why rename the mutated harness program?
Answer
Two modules with one PROGRAM-ID confuse the runtime, which finds the first
The second sed makes the program HARNESSB.
-
What does this program print?
IDENTIFICATION DIVISION. PROGRAM-ID. Q5. DATA DIVISION. WORKING-STORAGE SECTION. 01 N PIC 9(3) VALUE 0. 01 I PIC 9(2). PROCEDURE DIVISION. PERFORM VARYING I FROM 1 BY 1 UNTIL I IS GREATER THAN 3 ADD I TO N END-PERFORM. DISPLAY N. STOP RUN.Answer
006
One plus two plus three is six.
-
What does this program print?
IDENTIFICATION DIVISION. PROGRAM-ID. Q6. DATA DIVISION. WORKING-STORAGE SECTION. 01 A PIC 9(9) VALUE 10000. 01 B PIC 9(9) VALUE 6500. 01 C PIC S9(9). PROCEDURE DIVISION. COMPUTE C = A - B DISPLAY C. STOP RUN.Answer
+000003500
Ten thousand less six thousand five hundred is three thousand five hundred, positive.
-
What does this program print?
IDENTIFICATION DIVISION. PROGRAM-ID. Q7. DATA DIVISION. WORKING-STORAGE SECTION. 01 T. 05 T-C PIC X OCCURS 3 TIMES. PROCEDURE DIVISION. MOVE "ABC" TO T. INSPECT T REPLACING ALL "B" BY "X" DISPLAY T. STOP RUN.Answer
AXC
Only the B is replaced.
-
What does this program print?
IDENTIFICATION DIVISION. PROGRAM-ID. Q8. DATA DIVISION. WORKING-STORAGE SECTION. 01 V PIC 9(2) VALUE 2. PROCEDURE DIVISION. EVALUATE V WHEN 1 DISPLAY "one" WHEN 2 DISPLAY "two" WHEN OTHER DISPLAY "other" END-EVALUATE. STOP RUN.Answer
two
V is two, so the second WHEN runs.
-
Which of these does the harness NOT test?
Answer
The batch job, the file handling and the request parsing
The lesson says those need a golden-file test of their own.
-
Why does the lesson refuse to report a coverage number?
Answer
This image has no COBOL coverage tool, and a number from a tool that is not here would be invented
The page names the missing tools and stops there.
The code on this page was built with cobc (GnuCOBOL) 3.2.0 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container. The harness includes reqans.cpy, so its listing starts at the data division and the build commands on the page are the real ones.
Article author: Arthur Isaev