Parsing a CSV line
A comma-separated line is the simplest interface between two systems and the one with the most exceptions. This lesson parses one with UNSTRING, handles a field that is quoted and therefore holds a comma of its own, tests the amount before it is used, and prints the number of the line that failed.
UNSTRING splits on a comma
UNSTRING takes a field, a delimiter and a list of receivers. Every receiver takes the text up to the next delimiter, in order, and the text after the last delimiter goes into the last receiver:
IDENTIFICATION DIVISION.
PROGRAM-ID. SPLIT.
DATA DIVISION.
WORKING-STORAGE SECTION.
01 LINE-F PIC X(40) VALUE "42,Ada Lovelace,London,1250".
01 NAME-F PIC X(12).
01 CITY-F PIC X(10).
01 AMT-F PIC X(6).
01 ID-F PIC X(4).
01 ID-LEN PIC 99.
01 FIELDS-N PIC 9.
PROCEDURE DIVISION.
UNSTRING LINE-F DELIMITED BY ","
INTO ID-F COUNT IN ID-LEN
NAME-F
CITY-F
AMT-F
TALLYING IN FIELDS-N
END-UNSTRING.
DISPLAY "fields [" FIELDS-N "]".
DISPLAY "id [" ID-F "] len " ID-LEN.
DISPLAY "name [" NAME-F "]".
DISPLAY "city [" CITY-F "]".
DISPLAY "amount [" AMT-F "]".
STOP RUN.
cobc -x -free -Wall -o split unstring.cob
./split
fields [4]
id [42 ] len 02
name [Ada Lovelace]
city [London ]
amount [1250 ]
Each receiver is an ordinary field, so it has the length of its picture and the shorter values are padded with spaces on the right. The length of a piece is not visible in the receiving field; it is what COUNT IN is for.
The count and the tally
COUNT IN ID-LEN belongs to the receiver just before it and reports how many characters went into that receiver: two here, because the identifier in the line is 42. TALLYING IN FIELDS-N counts the receivers that were filled, four in this line.
Both numbers are what a parser needs in a report and in a check. The tally says whether the line had all the fields the program expects; the count says how long each field really was, which matters when the last field is optional or when the line arrives with a trailing delimiter. A line that ends with a comma fills four receivers and leaves the fourth one as an empty piece, so the tally alone does not prove the line is complete.
A comma inside quotes
A name can hold a comma, and then the writer of the file quotes it. A split on every comma is wrong for that line, and wrong in a way that moves every later field one place to the left:
IDENTIFICATION DIVISION.
PROGRAM-ID. QUOTED.
DATA DIVISION.
WORKING-STORAGE SECTION.
01 LINE-F PIC X(40) VALUE "42,""Lovelace, Ada"",London,1250".
01 PRE-F PIC X(20).
01 NAME-F PIC X(20).
01 POST-F PIC X(20).
01 DROP-F PIC X(2).
01 CITY-F PIC X(10).
01 AMT-F PIC X(6).
01 ID-F PIC X(4).
01 NAIVE-1 PIC X(10).
01 NAIVE-2 PIC X(10).
PROCEDURE DIVISION.
UNSTRING LINE-F DELIMITED BY ","
INTO NAIVE-1 NAIVE-2
END-UNSTRING.
DISPLAY "naive [" NAIVE-1 "] [" NAIVE-2 "]".
UNSTRING LINE-F DELIMITED BY QUOTE
INTO PRE-F NAME-F POST-F
END-UNSTRING.
DISPLAY "quoted [" NAME-F "]".
UNSTRING PRE-F DELIMITED BY "," INTO ID-F
END-UNSTRING.
UNSTRING POST-F DELIMITED BY "," INTO DROP-F CITY-F AMT-F
END-UNSTRING.
DISPLAY "id [" ID-F "] city [" CITY-F "] cents [" AMT-F "]".
STOP RUN.
cobc -x -free -Wall -o quoted quoted.cob
./quoted
naive [42 ] ["Lovelace ]
quoted [Lovelace, Ada ]
id [42 ] city [London ] cents [1250 ]
The naive split cut the quoted name in two and put a quote into the second piece. The second split uses the quote as the delimiter, so the three pieces are the text before the opening quote, the text between the quotes and the text after the closing quote, and the name comes out whole.
The rest is bookkeeping: the piece before the quote ends with a comma, and the piece after it starts with one, so the first split throws away DROP-F, the empty piece that the leading comma creates. A line with two quoted fields needs a loop, which is the subject of the lesson on PERFORM VARYING.
A field that is not a number
The amount is the field a program must not trust. It arrives as text, and once a receiver is padded with spaces the test CENTS-F IS NUMERIC fails on every value, because a space is not a digit. The test has to see the value without the padding:
01 CENTS-F PIC X(8) VALUE "1250".
01 CENTS-N PIC 9(8).
...
IF FUNCTION TRIM(CENTS-F) IS NUMERIC
MOVE FUNCTION TRIM(CENTS-F) TO CENTS-N
ELSE
DISPLAY "rejected, amount [" FUNCTION TRIM(CENTS-F) "]".
END-IF.
The block above is a fragment, not a program: the checker builds whole programs only, and the whole program is the next one. The test is the point. IS NUMERIC looks at every byte of what it is given, FUNCTION TRIM removes the padding, and the value that reaches the numeric field has been proved to be digits.
The whole report
Three lines, one paragraph, a running total and a message that names the line that failed:
IDENTIFICATION DIVISION.
PROGRAM-ID. CSVREP.
DATA DIVISION.
WORKING-STORAGE SECTION.
01 L1-F PIC X(40) VALUE "42,Ada Lovelace,London,1250".
01 L2-F PIC X(40) VALUE "43,Grace Hopper,New York,725".
01 L3-F PIC X(40) VALUE "44,Alan Turing,London,12x0".
01 LINE-F PIC X(40).
01 LINE-N PIC 9 VALUE 0.
01 GOOD-N PIC 9 VALUE 0.
01 BAD-N PIC 9 VALUE 0.
01 TOTAL-C PIC 9(7) VALUE 0.
01 ID-F PIC X(4).
01 NAME-F PIC X(12).
01 CITY-F PIC X(10).
01 CENTS-F PIC X(8).
01 CENTS-N PIC 9(8).
PROCEDURE DIVISION.
MOVE L1-F TO LINE-F.
PERFORM PARSE-LINE.
MOVE L2-F TO LINE-F.
PERFORM PARSE-LINE.
MOVE L3-F TO LINE-F.
PERFORM PARSE-LINE.
DISPLAY "good " GOOD-N " bad " BAD-N " total cents " TOTAL-C.
STOP RUN.
PARSE-LINE.
ADD 1 TO LINE-N.
UNSTRING LINE-F DELIMITED BY ","
INTO ID-F NAME-F CITY-F CENTS-F
END-UNSTRING.
IF FUNCTION TRIM(CENTS-F) IS NUMERIC
MOVE FUNCTION TRIM(CENTS-F) TO CENTS-N
ADD CENTS-N TO TOTAL-C
ADD 1 TO GOOD-N
DISPLAY "line " LINE-N ": " FUNCTION TRIM(ID-F)
" " FUNCTION TRIM(NAME-F)
" " FUNCTION TRIM(CITY-F)
" " CENTS-N " cents"
ELSE
ADD 1 TO BAD-N
DISPLAY "line " LINE-N ": rejected, amount ["
FUNCTION TRIM(CENTS-F) "]"
END-IF.
cobc -x -free -Wall -o csvrep report.cob
./csvrep
line 1: 42 Ada Lovelace London 00001250 cents
line 2: 43 Grace Hopper New York 00000725 cents
line 3: rejected, amount [12x0]
good 2 bad 1 total cents 0001975
The third line printed no total and did not stop the run: the bad line is counted, named and skipped, which is what a batch job must do with a file it did not write. The total is the sum of the two good lines and it is in cents, an integer, so no rounding happened anywhere.
Exercises
- Add a fifth field to the line and to the UNSTRING. What does TALLYING IN report when the line has only four?
- Print the length of every field with COUNT IN. Which field of the first line is shorter than its picture?
- Change the third line to 44,Alan Turing,London, so that the amount is empty. What does the program print, and why does TRIM not turn the empty field into zero?
- Split a line whose first character is a comma. How many receivers are filled, and what is in the first one?
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
What does UNSTRING do with the text after its last delimiter?
Answer
It goes into the last receiver
The receivers are filled left to right and whatever remains after the last delimiter belongs to the last one.
-
What does COUNT IN report after an UNSTRING?
Answer
The number of characters moved into the receiver before it
COUNT IN belongs to one receiver and gives the length of the piece it received; the number of filled receivers is what TALLYING IN counts.
-
A CSV line holds
42,"Lovelace, Ada",London. Why does a split on every comma fail?Answer
The comma inside the quotes splits the name and shifts every later field
The delimiter is found inside the quoted value, so the name is cut in two and the remaining pieces land in the wrong receivers.
-
Why does
CENTS-F IS NUMERICfail for a field that holds "1250" padded with spaces?Answer
IS NUMERIC tests every byte of the field, and a space is not a digit
The class test looks at the whole field, so the padding has to be removed first, which is what FUNCTION TRIM does.
-
What does this program print?
IDENTIFICATION DIVISION. PROGRAM-ID. Q5. DATA DIVISION. WORKING-STORAGE SECTION. 01 L-F PIC X(12) VALUE "AB,CD,EF". 01 A-F PIC X(3). 01 B-F PIC X(3). 01 C-F PIC X(3). PROCEDURE DIVISION. UNSTRING L-F DELIMITED BY "," INTO A-F B-F C-F END-UNSTRING. DISPLAY "[" A-F "][" B-F "][" C-F "]". STOP RUN.Answer
[AB ][CD ][EF ]
Each receiver is three bytes wide, so every piece is padded with a space.
-
What does this program print?
IDENTIFICATION DIVISION. PROGRAM-ID. Q6. DATA DIVISION. WORKING-STORAGE SECTION. 01 L-F PIC X(11) VALUE "AB,CD,EF". 01 A-F PIC X(3). 01 B-F PIC X(3). 01 N-F PIC 9. PROCEDURE DIVISION. UNSTRING L-F DELIMITED BY "," INTO A-F B-F TALLYING IN N-F END-UNSTRING. DISPLAY N-F " [" B-F "]". STOP RUN.Answer
2 [CD ]
Only two receivers are named, so two pieces are filled and the tally is two; the text after the second comma is not used.
-
What does this program print?
IDENTIFICATION DIVISION. PROGRAM-ID. Q7. DATA DIVISION. WORKING-STORAGE SECTION. 01 C-F PIC X(6) VALUE "12x0". 01 N-F PIC 9(4). PROCEDURE DIVISION. IF FUNCTION TRIM(C-F) IS NUMERIC DISPLAY "numeric" ELSE DISPLAY "not numeric" END-IF. STOP RUN.Answer
not numeric
The letter in the middle is not a digit, so the class test fails even after the padding is removed.
-
What does this program print?
IDENTIFICATION DIVISION. PROGRAM-ID. Q8. DATA DIVISION. WORKING-STORAGE SECTION. 01 L-F PIC X(10) VALUE ",AB". 01 A-F PIC X(3). 01 B-F PIC X(3). 01 N-F PIC 9. PROCEDURE DIVISION. UNSTRING L-F DELIMITED BY "," INTO A-F B-F TALLYING IN N-F END-UNSTRING. DISPLAY N-F " [" A-F "][" B-F "]". STOP RUN.Answer
2 [ ][AB ]
The line starts with the delimiter, so the first receiver gets an empty piece, the second gets AB, and both count.
-
What does this program print?
IDENTIFICATION DIVISION. PROGRAM-ID. Q9. DATA DIVISION. WORKING-STORAGE SECTION. 01 C-F PIC X(8) VALUE "1250". 01 N-F PIC 9(8). PROCEDURE DIVISION. MOVE FUNCTION TRIM(C-F) TO N-F. DISPLAY "[" N-F "]". STOP RUN.Answer
[00001250]
The receiving field holds eight digits, so the numeric move fills the left positions with zeros.
-
A batch job reads a CSV file and one amount holds a letter. What should the job do?
Answer
Reject that line, count it, and carry on with the rest of the file
The page rejects the bad line, names it and keeps going, so the good lines are still processed and the count of rejects is reported.
The code on this page was built with cobc (GnuCOBOL) 3.2.0 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.
Article author: Arthur Isaev