Capstone C: the test harness

The module has an interface, a version and three status codes. What it does not have is a way to prove that it still behaves after a change, and in this image there is no COBOL test framework to lean on: no cobol-check, no zUnit, no GnuCOBOL test runner. A COBOL program that calls the module and compares the answers is the whole of the tooling, and it is enough.

A harness in COBOL

A driver program holds a table of cases: a request and the answer it must produce. For each case it moves the request into the copybook record, CALLs the module, compares the status and the result, and counts. At the end it prints one line with the two counts and returns 1 if anything failed, which is the number a build script and a person both read.

The comparison is exact on purpose. A test that checks "no crash" passes when the module returns zero for every input, and a test that checks the status and the exact number of cents fails when the rounding changes by one. In a money program the exact number is the requirement.

The harness

The file holds two programs. The first is the harness: five cases in a table, a loop that calls the module with each and compares the status and the cents, and a summary that sets the exit status. The second is the module, the same one as in the previous lesson:

IDENTIFICATION DIVISION. PROGRAM-ID. HARNESS. DATA DIVISION. WORKING-STORAGE SECTION. 01 WS-CASES. 05 WS-CASE OCCURS 5 TIMES. 10 C-VERSION PIC 9(2). 10 C-OP PIC X(3). 10 C-CENTS PIC 9(9). 10 C-STATUS PIC 9(2). 10 C-EXPECT PIC S9(11). 01 WS-I PIC 9(2). 01 WS-PASSED PIC 9(3) VALUE 0. 01 WS-FAILED PIC 9(3) VALUE 0. 01 REQUEST-REC. 05 REQ-VERSION PIC 9(2). 05 REQ-OP PIC X(3). 05 REQ-CENTS PIC 9(9). 01 ANSWER-REC. 05 ANS-VERSION PIC 9(2). 05 ANS-STATUS PIC 9(2). 05 ANS-CENTS PIC S9(11). 05 ANS-MESSAGE PIC X(30). PROCEDURE DIVISION. MAIN-PARA. PERFORM FILL-CASES PERFORM VARYING WS-I FROM 1 BY 1 UNTIL WS-I > 5 PERFORM RUN-CASE END-PERFORM DISPLAY "passed " WS-PASSED " failed " WS-FAILED IF WS-FAILED IS GREATER THAN 0 MOVE 1 TO RETURN-CODE END-IF STOP RUN. FILL-CASES. MOVE 1 TO C-VERSION(1) MOVE "TAX" TO C-OP(1) MOVE 10000 TO C-CENTS(1) MOVE 0 TO C-STATUS(1) MOVE 1300 TO C-EXPECT(1) MOVE 1 TO C-VERSION(2) MOVE "TAX" TO C-OP(2) MOVE 25000 TO C-CENTS(2) MOVE 0 TO C-STATUS(2) MOVE 3250 TO C-EXPECT(2) MOVE 1 TO C-VERSION(3) MOVE "NET" TO C-OP(3) MOVE 50000 TO C-CENTS(3) MOVE 0 TO C-STATUS(3) MOVE 43500 TO C-EXPECT(3) MOVE 2 TO C-VERSION(4) MOVE "TAX" TO C-OP(4) MOVE 10000 TO C-CENTS(4) MOVE 8 TO C-STATUS(4) MOVE 0 TO C-EXPECT(4) MOVE 1 TO C-VERSION(5) MOVE "XXX" TO C-OP(5) MOVE 10000 TO C-CENTS(5) MOVE 9 TO C-STATUS(5) MOVE 0 TO C-EXPECT(5). RUN-CASE. MOVE C-VERSION(WS-I) TO REQ-VERSION MOVE C-OP(WS-I) TO REQ-OP MOVE C-CENTS(WS-I) TO REQ-CENTS CALL "TAXCALC" USING REQUEST-REC ANSWER-REC IF ANS-STATUS NOT = C-STATUS(WS-I) PERFORM FAIL-CASE ELSE IF ANS-CENTS NOT = C-EXPECT(WS-I) PERFORM FAIL-CASE ELSE ADD 1 TO WS-PASSED DISPLAY "pass case " WS-I END-IF END-IF. FAIL-CASE. ADD 1 TO WS-FAILED DISPLAY "FAIL case " WS-I " status " ANS-STATUS " cents " ANS-CENTS " expected " C-STATUS(WS-I) " " C-EXPECT(WS-I). END PROGRAM HARNESS. IDENTIFICATION DIVISION. PROGRAM-ID. TAXCALC. DATA DIVISION. WORKING-STORAGE SECTION. 01 WS-RATE PIC 9V99 VALUE 0.13. LINKAGE SECTION. 01 REQUEST-REC. 05 REQ-VERSION PIC 9(2). 05 REQ-OP PIC X(3). 05 REQ-CENTS PIC 9(9). 01 ANSWER-REC. 05 ANS-VERSION PIC 9(2). 05 ANS-STATUS PIC 9(2). 05 ANS-CENTS PIC S9(11). 05 ANS-MESSAGE PIC X(30). PROCEDURE DIVISION USING REQUEST-REC ANSWER-REC. MOVE 1 TO ANS-VERSION MOVE 0 TO ANS-CENTS MOVE SPACES TO ANS-MESSAGE IF REQ-VERSION NOT = 1 MOVE 8 TO ANS-STATUS MOVE "unsupported request version" TO ANS-MESSAGE GOBACK END-IF EVALUATE REQ-OP WHEN "TAX" COMPUTE ANS-CENTS ROUNDED = REQ-CENTS * WS-RATE MOVE 0 TO ANS-STATUS MOVE "tax calculated" TO ANS-MESSAGE WHEN "NET" COMPUTE ANS-CENTS = REQ-CENTS - 6500 MOVE 0 TO ANS-STATUS MOVE "net after fee" TO ANS-MESSAGE WHEN OTHER MOVE 9 TO ANS-STATUS MOVE "unknown operation" TO ANS-MESSAGE END-EVALUATE GOBACK. END PROGRAM TAXCALC.

The five cases cover the three status codes and the rounding: a tax of a hundred, a tax of two hundred and fifty, a net of five hundred, a request with version 2 and a request with an unknown operation. The expected cents are the numbers the module produced on the previous page, written down as constants, so a change in either side shows up as a failure rather than as a new output somebody has to notice.

The run that passes

cobc -x -free -Wall -o harness harness.cob
./harness
echo $?

pass case 01
pass case 02
pass case 03
pass case 04
pass case 05
passed 005 failed 000
0

Five passes and status 0, which is what a build script wants to see. The harness is built from the same file as the module, so the compiler and the test agree about the interface by construction: a record that changed in one program and not the other fails the build before it fails a test.

The run that fails

A harness nobody has seen fail is a harness nobody trusts. Change one expected value, one digit, and run it again:

sed 's/MOVE 1300 TO C-EXPECT(1)/MOVE 1350 TO C-EXPECT(1)/' \
    harness.cob > harness_bad.cob
sed -i 's/PROGRAM-ID. HARNESS./PROGRAM-ID. HARNESSB./; s/END PROGRAM HARNESS./END PROGRAM HARNESSB./' \
    harness_bad.cob
cobc -x -free -Wall -o harness_bad harness_bad.cob
./harness_bad
echo $?

FAIL case 01 status 00 cents +00000001300 expected 00 +00000001350
pass case 02
pass case 03
pass case 04
pass case 05
passed 004 failed 001
1

Four passes, one failure, status 1, and the failing line carries the actual answer, the expected one and the case number. That is the whole diagnostic a person needs, and it is on one line because a harness that prints a paragraph per failure is a harness whose output nobody reads.

The second sed renames the program in both places the name appears, the PROGRAM-ID and the END PROGRAM; a COBOL file whose END PROGRAM names a different program than its PROGRAM-ID is a compile error. Renaming is worth the two lines: two main programs with one name in one folder confuse the runtime, which resolves a name to the first it finds.

A harness in a suite

The harness is a program with a status, so it composes with everything that already understands statuses. A shell line runs it and stops on a failure:

cobc -x -free -Wall -o harness harness.cob taxcalc.cob || exit 2 ./harness || exit 1 echo "taxcalc: all cases passed"

Three properties make the pattern worth keeping. The exit status is the only interface a build server needs, so the harness can sit in a Makefile beside the production build. The cases are data in the program, so adding one is a five-line change and not a new framework. And the harness is plain COBOL, so the person who maintains the module can read and extend it on the day the module changes, which is not true of a test written in a language they do not use.

What the harness does not do is as important. It does not test the batch job of the previous lesson, the file handling or the parsing of the request line; those need a program that reads a file and compares its output, which is a golden-file test and a topic of its own. It does not measure coverage: this image has no COBOL coverage tool, and a number produced by a tool that does not exist here would be a number invented.

Capstone C ends here: a contract in a copybook, a module that implements it, a batch job that uses it and a harness that proves it. The three capstones together are the course in miniature: a file layout, an index and a report in A and B, an interface and its tests in C.

Exercises

  1. Add a case for an amount of one cent. What does the module answer, and is the rounding what you expected?
  2. Add a case with an empty operation. Which status does it get, and which side of the interface decides that?
  3. Make the harness print the failing request as well as the answer. Which fields do you add to the line?
  4. Change the module\'s rate to 0.20 and run the harness without touching the cases. How many fail, and what does that say about the table?

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. What does the harness return when a case fails?

    Answer

    1

    The failed run printed passed 004 failed 001 and status 1.

  2. Why compare the exact number of cents?

    Answer

    In a money program the exact result is the requirement, and a changed round must fail

    The harness checks ANS-STATUS and ANS-CENTS exactly.

  3. What does the failing line of the mutated run show?

    Answer

    The case number, the actual status and cents, and the expected ones

    FAIL case 01 status 00 cents +00000001300 expected 00 +00000001350.

  4. Why rename the mutated harness program?

    Answer

    Two modules with one PROGRAM-ID confuse the runtime, which finds the first

    The second sed makes the program HARNESSB.

  5. What does this program print? IDENTIFICATION DIVISION. PROGRAM-ID. Q5. DATA DIVISION. WORKING-STORAGE SECTION. 01 N PIC 9(3) VALUE 0. 01 I PIC 9(2). PROCEDURE DIVISION. PERFORM VARYING I FROM 1 BY 1 UNTIL I IS GREATER THAN 3 ADD I TO N END-PERFORM. DISPLAY N. STOP RUN.

    Answer

    006

    One plus two plus three is six.

  6. What does this program print? IDENTIFICATION DIVISION. PROGRAM-ID. Q6. DATA DIVISION. WORKING-STORAGE SECTION. 01 A PIC 9(9) VALUE 10000. 01 B PIC 9(9) VALUE 6500. 01 C PIC S9(9). PROCEDURE DIVISION. COMPUTE C = A - B DISPLAY C. STOP RUN.

    Answer

    +000003500

    Ten thousand less six thousand five hundred is three thousand five hundred, positive.

  7. What does this program print? IDENTIFICATION DIVISION. PROGRAM-ID. Q7. DATA DIVISION. WORKING-STORAGE SECTION. 01 T. 05 T-C PIC X OCCURS 3 TIMES. PROCEDURE DIVISION. MOVE "ABC" TO T. INSPECT T REPLACING ALL "B" BY "X" DISPLAY T. STOP RUN.

    Answer

    AXC

    Only the B is replaced.

  8. What does this program print? IDENTIFICATION DIVISION. PROGRAM-ID. Q8. DATA DIVISION. WORKING-STORAGE SECTION. 01 V PIC 9(2) VALUE 2. PROCEDURE DIVISION. EVALUATE V WHEN 1 DISPLAY "one" WHEN 2 DISPLAY "two" WHEN OTHER DISPLAY "other" END-EVALUATE. STOP RUN.

    Answer

    two

    V is two, so the second WHEN runs.

  9. Which of these does the harness NOT test?

    Answer

    The batch job, the file handling and the request parsing

    The lesson says those need a golden-file test of their own.

  10. Why does the lesson refuse to report a coverage number?

    Answer

    This image has no COBOL coverage tool, and a number from a tool that is not here would be invented

    The page names the missing tools and stops there.

The code on this page was built with cobc (GnuCOBOL) 3.2.0 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container. The harness includes reqans.cpy, so its listing starts at the data division and the build commands on the page are the real ones.

Article author: Arthur Isaev