Runtime checks with -debug

A test suite answers one question about a change: does the program still do what it did. It answers a second question almost by accident, and only if the build that runs the tests is a checked build: does the program do anything the language does not allow. This page is that second question, with a bug that passes every test in an unchecked build.

A loop one entry too long

The file is the one of the previous page with one change in the unit: the rates come from a table of three and are copied into the rate table of two. The author added a third rate, 0.07, and forgot that the table it goes into has two entries. The harness keeps its four cases:

IDENTIFICATION DIVISION. PROGRAM-ID. HARNNESSB. DATA DIVISION. WORKING-STORAGE SECTION. 01 WS-CASES. 05 WS-CASE OCCURS 4 TIMES. 10 C-CODE PIC X(1). 10 C-AMOUNT PIC 9(7)V99. 10 C-STATUS PIC 9(2). 10 C-DISCOUNT PIC 9(7)V99. 01 WS-I PIC 9(2). 01 WS-PASSED PIC 9(3) VALUE 0. 01 WS-FAILED PIC 9(3) VALUE 0. 01 WS-OUT PIC Z(6)9.99. 01 DISCOUNT-REQUEST. 05 DR-CODE PIC X(1). 05 DR-AMOUNT PIC 9(7)V99. 01 DISCOUNT-ANSWER. 05 DA-STATUS PIC 9(2). 05 DA-DISCOUNT PIC 9(7)V99. 05 DA-MESSAGE PIC X(30). PROCEDURE DIVISION. MAIN-PARA. PERFORM FILL-CASES PERFORM VARYING WS-I FROM 1 BY 1 UNTIL WS-I > 4 PERFORM RUN-CASE END-PERFORM DISPLAY "passed " WS-PASSED " failed " WS-FAILED IF WS-FAILED IS GREATER THAN 0 MOVE 1 TO RETURN-CODE END-IF STOP RUN. FILL-CASES. MOVE "A" TO C-CODE(1) MOVE 100.00 TO C-AMOUNT(1) MOVE 0 TO C-STATUS(1) MOVE 10.00 TO C-DISCOUNT(1) MOVE "A" TO C-CODE(2) MOVE 250.00 TO C-AMOUNT(2) MOVE 0 TO C-STATUS(2) MOVE 25.00 TO C-DISCOUNT(2) MOVE "B" TO C-CODE(3) MOVE 100.00 TO C-AMOUNT(3) MOVE 0 TO C-STATUS(3) MOVE 5.00 TO C-DISCOUNT(3) MOVE "X" TO C-CODE(4) MOVE 100.00 TO C-AMOUNT(4) MOVE 9 TO C-STATUS(4) MOVE 0 TO C-DISCOUNT(4). RUN-CASE. MOVE C-CODE(WS-I) TO DR-CODE MOVE C-AMOUNT(WS-I) TO DR-AMOUNT CALL "DISCOUNTB" USING DISCOUNT-REQUEST DISCOUNT-ANSWER IF DA-STATUS NOT = C-STATUS(WS-I) PERFORM FAIL-CASE ELSE IF DA-DISCOUNT NOT = C-DISCOUNT(WS-I) PERFORM FAIL-CASE ELSE ADD 1 TO WS-PASSED DISPLAY "pass case " WS-I END-IF END-IF. FAIL-CASE. ADD 1 TO WS-FAILED MOVE DA-DISCOUNT TO WS-OUT DISPLAY "FAIL case " WS-I " status " DA-STATUS " discount " FUNCTION TRIM(WS-OUT) " expected " C-STATUS(WS-I) " " C-DISCOUNT(WS-I). IDENTIFICATION DIVISION. PROGRAM-ID. DISCOUNTB. DATA DIVISION. WORKING-STORAGE SECTION. 01 RATE-SOURCE. 05 RATE-SRC PIC 9V99 OCCURS 3 TIMES. 01 RATE-TABLE. 05 RATE-ENTRY PIC 9V99 OCCURS 2 TIMES. 01 WS-INDEX PIC 9. 01 WS-I PIC 9. 01 WS-RATE PIC 9V99. LINKAGE SECTION. 01 DISCOUNT-REQUEST. 05 DR-CODE PIC X(1). 05 DR-AMOUNT PIC 9(7)V99. 01 DISCOUNT-ANSWER. 05 DA-STATUS PIC 9(2). 05 DA-DISCOUNT PIC 9(7)V99. 05 DA-MESSAGE PIC X(30). PROCEDURE DIVISION USING DISCOUNT-REQUEST DISCOUNT-ANSWER. MOVE 0 TO DA-STATUS MOVE 0 TO DA-DISCOUNT MOVE SPACES TO DA-MESSAGE MOVE 0.10 TO RATE-SRC(1) MOVE 0.05 TO RATE-SRC(2) MOVE 0.07 TO RATE-SRC(3) PERFORM VARYING WS-I FROM 1 BY 1 UNTIL WS-I IS GREATER THAN 3 MOVE RATE-SRC(WS-I) TO RATE-ENTRY(WS-I) END-PERFORM EVALUATE DR-CODE WHEN "A" MOVE 1 TO WS-INDEX WHEN "B" MOVE 2 TO WS-INDEX WHEN OTHER MOVE 9 TO DA-STATUS MOVE "unknown discount code" TO DA-MESSAGE GOBACK END-EVALUATE MOVE RATE-ENTRY(WS-INDEX) TO WS-RATE COMPUTE DA-DISCOUNT ROUNDED = DR-AMOUNT * WS-RATE GOBACK. END PROGRAM DISCOUNTB.

The loop is the bug: UNTIL WS-I IS GREATER THAN 3 over a table declared OCCURS 2 TIMES. A person reading the declaration can see it, which is the best case; the other case is a count that comes from a file, and then nothing in the source says which way the loop will end.

The run whose tests all pass

Built the way the suite builds it, without -debug, the file passes all four cases:

cobc -x -free -Wall -o harness_bug harness_bug.cob
./harness_bug
echo $?

pass case 01
pass case 02
pass case 03
pass case 04
passed 004 failed 000
0

Four passes and status 0. The third rate went into the two bytes after the table, which happen to be another working-storage field, and nothing about the answers changed: the cases do not read the clobbered field, and the write itself is invisible. A test suite run only on this build would sign the change off.

That is the whole argument for checked builds in tests, and it is not theoretical: this is a real run of a real bug, and every line of it is on the page. The unchecked build is not wrong, it is doing what the language allows; the program is wrong, and only the checked build says so.

The checked run

The same source and the same four cases, built with -debug so that the runtime checks every subscript:

cobc -x -free -Wall -debug -o harness_dbg harness_bug.cob
./harness_dbg
echo $?

libcob: harness_bug.cob:106: error: subscript of 'RATE-ENTRY' out of bounds: 3
note: maximum subscript for 'RATE-ENTRY': 2

 Last statement of "DISCOUNTB" was MOVE at line 106 of harness_bug.cob
    ENTRY DISCOUNTB at harness_bug.cob:99
 Last statement of "HARNNESSB" was CALL
    RUN-CASE at harness_bug.cob:57
    MAIN-PARA at harness_bug.cob:28
    ENTRY HARNNESSB at harness_bug.cob:25
 Started by ./harness_dbg
1

The message names the field, the line and the maximum, and the PERFORM stack below it names the harness and the paragraph that called the unit. That last part is why a checked run is worth having in a suite: the failure arrives with the test that found it, not with a stack trace from production a week later.

Putting -debug in the test build

The test build is a second target in the same build file, with the same source and one more option:

COBC = cobc FLAGS = -free -Wall test: harness_dbg
    ./harness_dbg

harness_dbg: harness_bug.cob
    $(COBC) -x $(FLAGS) -debug -o harness_dbg harness_bug.cob

The production build keeps the plain flags, because the checks cost time on every statement; the test target pays that time and gets the message. Two targets, one source, and a rule that the test target is the one the suite runs.

What the checks cost

The lesson on -debug measures it on a loop of arithmetic: about a fifth more wall time, 246.7 against 299.1 milliseconds for twenty million additions. A test suite of a few hundred calls pays that fifth and gets the message above; a nightly job of a million records does not, which is why the two builds exist.

The next page takes the other half of testing a program like this one: the output of a report, compared with a file that holds the output somebody approved, line by line.

The code on this page was built with cobc (GnuCOBOL) 3.2.0 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev