Capstone A: reading and validating
A fixed-width file from anywhere in the world arrives with records that do not fit the layout: a name cut short by a source system, a region code added last year, a line a person edited by hand. The batch program that reads it has to say which record is wrong and why, and it has to fail the step so somebody looks.
Reading the records
The program opens the file, and because it is also the program that makes the example runnable, it writes the eight sample records when the file is not there. The status of the first OPEN INPUT is 35 in that case, and the seed paragraph writes the file the page shows:
IDENTIFICATION DIVISION.
PROGRAM-ID. VALIDATE.
ENVIRONMENT DIVISION.
INPUT-OUTPUT SECTION.
FILE-CONTROL.
SELECT CUS-F ASSIGN TO "customers.dat"
ORGANIZATION IS LINE SEQUENTIAL
FILE STATUS IS WS-STATUS.
DATA DIVISION.
FILE SECTION.
FD CUS-F.
01 CUSTOMER-REC.
05 CUST-ID PIC X(6).
05 CUST-NAME PIC X(20).
05 CUST-REGION PIC X(2).
05 CUST-AMOUNT PIC 9(7)V99.
WORKING-STORAGE SECTION.
01 WS-STATUS PIC XX.
01 WS-EOF PIC X VALUE "N".
01 WS-PREV PIC X(6) VALUE SPACES.
01 WS-ACCEPTED PIC 9(3) VALUE 0.
01 WS-REJECTED PIC 9(3) VALUE 0.
01 WS-REASON PIC X(30).
01 WS-SEED-LINE PIC X(37).
01 WS-OUT PIC Z(6)9.99.
PROCEDURE DIVISION.
MAIN-PARA.
OPEN INPUT CUS-F
IF WS-STATUS = "35"
PERFORM SEED-DATA
END-IF
CLOSE CUS-F
OPEN INPUT CUS-F
PERFORM UNTIL WS-EOF = "Y"
READ CUS-F
AT END MOVE "Y" TO WS-EOF
NOT AT END
PERFORM CHECK-RECORD
END-READ
END-PERFORM
CLOSE CUS-F
DISPLAY "accepted " WS-ACCEPTED " rejected " WS-REJECTED
IF WS-REJECTED IS GREATER THAN 0
MOVE 4 TO RETURN-CODE
END-IF
STOP RUN.
SEED-DATA.
OPEN OUTPUT CUS-F
MOVE "C00001Ada Lovelace N1000123456" TO WS-SEED-LINE
WRITE CUSTOMER-REC FROM WS-SEED-LINE
MOVE "C00002Grace Hopper N1000234567" TO WS-SEED-LINE
WRITE CUSTOMER-REC FROM WS-SEED-LINE
MOVE "C00003Alan Turing S2000045678" TO WS-SEED-LINE
WRITE CUSTOMER-REC FROM WS-SEED-LINE
MOVE "C00004Katherine Johnson S2000567890" TO WS-SEED-LINE
WRITE CUSTOMER-REC FROM WS-SEED-LINE
MOVE "C00005Edsger Dijkstra E3000078901" TO WS-SEED-LINE
WRITE CUSTOMER-REC FROM WS-SEED-LINE
MOVE "C00006Barbara Liskov E3000123456" TO WS-SEED-LINE
WRITE CUSTOMER-REC FROM WS-SEED-LINE
MOVE "C00007Bad Region X9000010000" TO WS-SEED-LINE
WRITE CUSTOMER-REC FROM WS-SEED-LINE
MOVE "C00008Short " TO WS-SEED-LINE
WRITE CUSTOMER-REC FROM WS-SEED-LINE
CLOSE CUS-F.
CHECK-RECORD.
MOVE SPACES TO WS-REASON
IF CUST-ID(1:1) NOT = "C"
MOVE "id does not start with C" TO WS-REASON
ELSE
IF CUST-ID = WS-PREV
MOVE "duplicate id" TO WS-REASON
END-IF
END-IF
IF WS-REASON = SPACES
IF CUST-AMOUNT IS NOT NUMERIC
MOVE "amount is not numeric" TO WS-REASON
ELSE
IF CUST-AMOUNT = 0
MOVE "amount is zero" TO WS-REASON
END-IF
END-IF
END-IF
IF WS-REASON = SPACES
EVALUATE CUST-REGION
WHEN "N1" CONTINUE
WHEN "S2" CONTINUE
WHEN "E3" CONTINUE
WHEN OTHER MOVE "unknown region" TO WS-REASON
END-EVALUATE
END-IF
MOVE CUST-ID TO WS-PREV
IF WS-REASON = SPACES
MOVE CUST-AMOUNT TO WS-OUT
ADD 1 TO WS-ACCEPTED
DISPLAY "accept " CUST-ID " " FUNCTION TRIM(CUST-REGION)
" " FUNCTION TRIM(WS-OUT)
ELSE
ADD 1 TO WS-REJECTED
DISPLAY "reject " CUST-ID " " FUNCTION TRIM(WS-REASON)
END-IF.
Above the listing stand IDENTIFICATION DIVISION. and PROGRAM-ID. VALIDATE.. The FD is the record of the previous lesson, with the same four fields in the same order: that is the whole contract between the file and the program.
The validation paragraph
CHECK-RECORD writes one reason into WS-REASON and tests it after every rule, so the first rule that fails is the one reported and the later rules do not overwrite it. Four rules, in the order that gives the most useful message:
- The id must start with C, and it must differ from the previous record, which catches a duplicate in a file sorted by id.
- The amount must be a number. A short line has spaces in the amount, and IS NOT NUMERIC is the test that finds it.
- The amount must not be zero, which is a valid number and usually a missing value.
- The region must be one of the three the business uses, which an EVALUATE with WHEN OTHER checks without a chain of IFs.
The order matters for the person reading the log. A record with a bad region and a bad amount reports the amount, because a record whose amount cannot be read is not worth discussing further; a record with a region that is new to the program is reported as such even though its amount is fine.
The run and its status
cobc -x -free -Wall -o validate validate.cob
./validate
echo $?
accept C00001 N1 1234.56
accept C00002 N1 2345.67
accept C00003 S2 456.78
accept C00004 S2 5678.90
accept C00005 E3 789.01
accept C00006 E3 1234.56
reject C00007 unknown region
reject C00008 amount is not numeric
accepted 006 rejected 002
4
Six accepted, two rejected, status 4. The number is the one from the batch patterns lesson: 4 is a warning that a scheduler may choose to let through, and it is a different kind of failure from the 8 of a missing file. A job that must not continue with rejected records returns 8 instead, or writes the rejects to a file and returns 4 while the file is empty.
Two rejects, two reasons
The two bad lines of the data file are different mistakes and the program reports them differently. C00007 is a thirty-seven-byte record with a region of X9: the layout is right, the business rule is not. The fix is either a new region in the EVALUATE or a corrected record.
C00008 is eleven bytes, not thirty-seven. The line sequential runtime pads the record with spaces to the length of the FD, so the region becomes two spaces and the amount becomes nine spaces, and the numeric test fails before the region test ever runs. The reason printed is therefore about the amount, which is the symptom; the cause is a line that was never a record.
That difference is worth a line in the log. A reject count of two says nothing about whether the source system changed its layout or one row was mistyped, and the next person to read the job at three in the morning needs to know which.
What a job does with a reject
A real validation step writes the good records to one file and the bad ones to another, with the reason appended, and prints the counts. The shape is one more FD, one OPEN OUTPUT for the reject file, and a WRITE in the branch that already prints the reject line. The status stays 4, the reject file is what a person reads, and the good file is what the next step processes.
Two rules keep such a step honest. The counts must add up to the number of records read: accepted + rejected = read, printed in the trailer, is the cheapest check that no record was dropped by a bug in the loop. And a reject file that is not empty must be named in the message, so the scheduler log points at the file rather than at the step.
Exercises
- Add the check that the id is six characters and starts with a letter. Which record of the sample data fails it?
- Write the rejects to rejects.dat with the reason after the record. Which FD do you need, and why is it longer than thirty-seven bytes?
- Change the order of the rules so the region is checked first. Which reason does C00008 report now, and why is that worse?
- Make the program return 8 when the reject count is more than a tenth of the records read. Which status does the sample data produce?
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
Which status does the validation run return?
Answer
4, a warning, because two records were rejected
The run ends with 4 and the last line says accepted 006 rejected 002.
-
Why is C00008 rejected for its amount rather than its region?
Answer
The short line is padded with spaces, and the amount test runs before the region test
Nine spaces are not a number, and the amount rule comes first.
-
What does IS NOT NUMERIC find?
Answer
A numeric field whose bytes are not all digits
The padded amount holds spaces, and the test rejects it.
-
Which check proves that the loop dropped no record?
Answer
accepted plus rejected equals the number read
The lesson recommends that line in the trailer.
-
What does this program print?
IDENTIFICATION DIVISION. PROGRAM-ID. Q5. DATA DIVISION. WORKING-STORAGE SECTION. 01 A PIC X(3) VALUE "C01". 01 B PIC X(3) VALUE "C01". PROCEDURE DIVISION. IF A = B DISPLAY "same" ELSE DISPLAY "different" END-IF. STOP RUN.Answer
same
Both fields hold the same three bytes.
-
What does this program print?
IDENTIFICATION DIVISION. PROGRAM-ID. Q6. DATA DIVISION. WORKING-STORAGE SECTION. 01 A PIC 9(5) VALUE 0. 01 T PIC 9(5). PROCEDURE DIVISION. MOVE "12x45" TO T IF T IS NOT NUMERIC DISPLAY "bad" ELSE DISPLAY "good" END-IF. STOP RUN.Answer
bad
The x is not a digit, so the numeric test fails.
-
What does this program print?
IDENTIFICATION DIVISION. PROGRAM-ID. Q7. DATA DIVISION. WORKING-STORAGE SECTION. 01 A PIC 9(5)V99 VALUE 1234.56. 01 B PIC 9(3)V99 VALUE 2. 01 C PIC 9(6)V99. PROCEDURE DIVISION. COMPUTE C = A + B DISPLAY C. STOP RUN.Answer
01236.56
A thousand two hundred and thirty-four plus two is a thousand two hundred and thirty-six.
-
What does this program print?
IDENTIFICATION DIVISION. PROGRAM-ID. Q8. DATA DIVISION. WORKING-STORAGE SECTION. 01 R PIC X(6) VALUE "C00001". PROCEDURE DIVISION. DISPLAY R(1:1) " " R(2:5). STOP RUN.Answer
C 00001
One character from the start, then five from position two.
-
Why keep the region list in an EVALUATE rather than four IFs?
Answer
One WHEN OTHER catches every unknown code and adds a region in one line
The validation uses EVALUATE with three known regions and WHEN OTHER.
-
Why does a short line change the meaning of the last fields?
Answer
The reader pads the record with spaces to the declared length
C00008 became eleven bytes plus twenty-six spaces.
The code on this page was built with cobc (GnuCOBOL) 3.2.0 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev