ASCII, EBCDIC and UTF-8

A COBOL field of characters is a run of bytes, and the bytes are numbers: a letter is a number, a space is a number, a digit is a number. The language hardly ever shows those numbers, which is why a file written on one machine and read on another can be wrong in a way no statement reports.

A character is a byte

FUNCTION ORD returns the position of a character in the program collating sequence, and FUNCTION CHAR goes back the other way:

IDENTIFICATION DIVISION. PROGRAM-ID. ORDCH. DATA DIVISION. WORKING-STORAGE SECTION. 01 A-F PIC X VALUE "A". 01 Z-F PIC X VALUE "Z". 01 N-F PIC X VALUE "0". 01 S-F PIC X VALUE SPACE. PROCEDURE DIVISION. DISPLAY "A = " FUNCTION ORD(A-F). DISPLAY "Z = " FUNCTION ORD(Z-F). DISPLAY "0 = " FUNCTION ORD(N-F). DISPLAY "space = " FUNCTION ORD(S-F). DISPLAY "49 = [" FUNCTION CHAR(49) "]". DISPLAY "233 round trip = " FUNCTION ORD(FUNCTION CHAR(233)). STOP RUN.

cobc -x -free -Wall -o ordch ordch.cob
./ordch

A = 000000066
Z = 000000091
0 = 000000049
space = 000000033
49 = [0]
233 round trip = 000000233

The numbers are one more than the ASCII codes this platform stores: A is byte 65 and FUNCTION ORD reports 66, because a collating sequence is numbered from one and not from zero. A program that needs the byte value itself subtracts one, and a program that only compares characters never needs the number at all.

The nine-digit result is the width of the function, not a choice of the program: ORD returns an integer large enough for every position the compiler supports, and DISPLAY writes it in that width.

UTF-8: a letter is two bytes

The source file of this page is UTF-8, and a literal with an accented letter puts its bytes into the field. A COBOL text field counts bytes and not characters:

IDENTIFICATION DIVISION. PROGRAM-ID. UTF8. DATA DIVISION. WORKING-STORAGE SECTION. 01 WORD-F PIC X(6) VALUE "café". 01 B1-F PIC X. 01 B2-F PIC X. PROCEDURE DIVISION. DISPLAY "[" WORD-F "]". DISPLAY "length " LENGTH OF WORD-F. DISPLAY "literal length " LENGTH OF "café". MOVE WORD-F(4:1) TO B1-F. MOVE WORD-F(5:1) TO B2-F. DISPLAY "byte 4 = " FUNCTION ORD(B1-F). DISPLAY "byte 5 = " FUNCTION ORD(B2-F). STOP RUN.

cobc -x -free -Wall -o utf8 utf8.cob
./utf8

[café ]
length 6
literal length 5
byte 4 = 000000196
byte 5 = 000000170

Four letters and five bytes: é arrived as the two bytes 195 and 169, which FUNCTION ORD reports as positions 196 and 170. LENGTH OF the literal is 5, and the field of six bytes has one space of padding, so the run prints five characters and one space between the brackets. Nothing on the page or in the compiler counted four characters, because the language has no idea that the two bytes belong together.

A cut at a byte, not a character

Reference modification counts bytes. Bytes four and five of WORD-F are the two halves of the letter, and the program above took them apart without an error. The bytes of the whole output show it:

./utf8 | hexdump -C

00000000 5b 63 61 66 c3 a9 20 5d 0a 6c 65 6e 67 74 68 20 |[caf.. ].length |
00000010 36 0a 6c 69 74 65 72 61 6c 20 6c 65 6e 67 74 68 |6.literal length|
00000020 20 35 0a 62 79 74 65 20 34 20 3d 20 30 30 30 30 | 5.byte 4 = 0000|
00000030 30 30 31 39 36 0a 62 79 74 65 20 35 20 3d 20 30 |00196.byte 5 = 0|
00000040 30 30 30 30 30 31 37 30 0a |00000170.|

c3 a9 is the letter, and the next byte 20 is the padding space. A program that cuts a field at a fixed position is safe on ASCII text and unsafe on a text that may hold a non-ASCII letter, and the language will not warn: the reference modification is legal for every position inside the field.

EBCDIC and the collating sequence

A mainframe stores characters in EBCDIC, where the code of a is larger than the code of A in a different order from ASCII. COBOL names the order a program compares in, and PROGRAM COLLATING SEQUENCE chooses it:

IDENTIFICATION DIVISION. PROGRAM-ID. COLL. ENVIRONMENT DIVISION. CONFIGURATION SECTION. SPECIAL-NAMES. ALPHABET EBC IS EBCDIC. OBJECT-COMPUTER. PROGRAM COLLATING SEQUENCE IS EBC. DATA DIVISION. WORKING-STORAGE SECTION. 01 A-F PIC X VALUE "A". 01 L-F PIC X VALUE "a". PROCEDURE DIVISION. DISPLAY "A = " FUNCTION ORD(A-F). DISPLAY "a = " FUNCTION ORD(L-F). IF L-F < A-F DISPLAY "a sorts before A" ELSE DISPLAY "A sorts before a" END-IF. STOP RUN.

cobc -x -free -Wall -o coll coll.cob
./coll

A = 000000066
a = 000000098
a sorts before A

The comparison followed the EBCDIC order, where the lower case letters come before the upper case ones, while FUNCTION ORD still reported the ASCII positions of the two bytes. On this platform the alphabet changes the order of comparisons and does not change the bytes in memory, so a sorted key is EBCDIC-ordered while the file it goes into is still ASCII.

That difference is the one a program moved from z/OS to a Linux server meets first. The page on the mainframe context says where the encoding is handled in that world and what this course does not run.

CODE-SET and the bytes of a file

A file can declare the encoding of its records with CODE-SET on its FD. The clause is accepted here and does not convert anything yet: the compiler says so itself.

IDENTIFICATION DIVISION. PROGRAM-ID. CODESET. ENVIRONMENT DIVISION. CONFIGURATION SECTION. SPECIAL-NAMES. ALPHABET EBC IS EBCDIC. INPUT-OUTPUT SECTION. FILE-CONTROL. SELECT OUT-F ASSIGN TO "ebc.dat" ORGANIZATION IS LINE SEQUENTIAL. DATA DIVISION. FILE SECTION. FD OUT-F CODE-SET IS EBC. 01 REC-F PIC X(5). PROCEDURE DIVISION. OPEN OUTPUT OUT-F. MOVE "AB1" TO REC-F. WRITE REC-F. CLOSE OUT-F. STOP RUN.

cobc -x -free -Wall -o codeset codeset.cob
./codeset
hexdump -C ebc.dat

codeset.cob:16: warning: handling of CODE-SET is unfinished; implementation is likely to be changed [-Wunfinished]
14 | FILE SECTION.
15 | FD OUT-F
16 > CODE-SET IS EBC.
17 | 01 REC-F PIC X(5).
18 |
00000000 c1 c2 f1 0a |....|

c1 c2 f1 is ASCII AB1, not the EBCDIC bytes of the three characters, and the file ends with the line-sequence newline. The warning is the honest state of the clause in version 3.2.0: the statement compiles, the conversion does not happen, and the lesson on line sequential files writes and reads the same kind of file without it.

This is also where the part on text ends. The next part is control flow, which starts with the IF that every one of these tests has been using already.

Exercises

  1. Print the position of the digits 0 to 9 with FUNCTION ORD. What is the difference between two neighbours?
  2. Build a string from FUNCTION CHAR with the positions 66, 67 and 68. Which three letters do you get?
  3. Read a text of three letters with a reference modification of one byte at a time and print each position. Which one is changed by the EBCDIC collating sequence?
  4. Change WORD-F to PIC X(4) VALUE "café" and see what the compiler says. Which warning names the truncation?

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. What does FUNCTION ORD return for the letter A?

    Answer

    Its position in the collating sequence, one more than the ASCII code

    A collating sequence is numbered from one, and the run on this page reports 66 for the byte 65.

  2. How many bytes does the literal "café" occupy in a UTF-8 source?

    Answer

    Five: four letters, and the accented one takes two bytes

    LENGTH OF the literal is 5 on this page, because UTF-8 writes the accented letter as two bytes.

  3. What does a reference modification count?

    Answer

    Bytes, so a UTF-8 letter can be cut in half

    The language has no character type beyond the byte, so the positions are byte positions.

  4. What does PROGRAM COLLATING SEQUENCE change?

    Answer

    The order in which text values compare, not the bytes in memory

    The run compares a with A in EBCDIC order while FUNCTION ORD still reports the ASCII positions.

  5. What does this program print? IDENTIFICATION DIVISION. PROGRAM-ID. Q5. DATA DIVISION. WORKING-STORAGE SECTION. 01 T-F PIC X(5) VALUE "café". PROCEDURE DIVISION. DISPLAY FUNCTION LENGTH("café"). STOP RUN.

    Answer

    5

    Four letters in five bytes: the accented letter is two bytes in UTF-8 and LENGTH counts bytes.

  6. What does this program print? IDENTIFICATION DIVISION. PROGRAM-ID. Q6. DATA DIVISION. WORKING-STORAGE SECTION. 01 S-F PIC X VALUE SPACE. PROCEDURE DIVISION. DISPLAY FUNCTION ORD(S-F). STOP RUN.

    Answer

    000000033

    A space is byte 32 in ASCII, and the collating position is one more.

  7. What does this program print? IDENTIFICATION DIVISION. PROGRAM-ID. Q7. DATA DIVISION. WORKING-STORAGE SECTION. 01 C-F PIC X. PROCEDURE DIVISION. MOVE FUNCTION CHAR(66) TO C-F. DISPLAY "[" C-F "]". STOP RUN.

    Answer

    [A]

    CHAR takes a position in the collating sequence, so 66 is the letter A and not the byte 66.

  8. What does this program print? IDENTIFICATION DIVISION. PROGRAM-ID. Q8. ENVIRONMENT DIVISION. CONFIGURATION SECTION. SPECIAL-NAMES. ALPHABET EBC IS EBCDIC. OBJECT-COMPUTER. PROGRAM COLLATING SEQUENCE IS EBC. DATA DIVISION. WORKING-STORAGE SECTION. 01 U-F PIC X VALUE "A". 01 L-F PIC X VALUE "a". PROCEDURE DIVISION. IF L-F < U-F DISPLAY "lower first" ELSE DISPLAY "upper first" END-IF. STOP RUN.

    Answer

    lower first

    In the EBCDIC order the lower case letters sort before the upper case ones, so a is less than A.

  9. What does this program print? IDENTIFICATION DIVISION. PROGRAM-ID. Q9. DATA DIVISION. WORKING-STORAGE SECTION. 01 T-F PIC X(3). PROCEDURE DIVISION. MOVE "café" TO T-F. DISPLAY "[" T-F "]". STOP RUN.

    Answer

    [caf]

    The field holds three bytes, so the four letters do not fit and only the first three bytes are kept; the compiler also warns about the truncation.

  10. A file from a mainframe is full of EBCDIC bytes. What will a Linux program that reads it as text show?

    Answer

    Wrong characters, because the byte values are read as ASCII

    Nothing converts the bytes on this platform; CODE-SET builds with a warning that its handling is unfinished, and the file has to be converted before it is read.

The code on this page was built with cobc (GnuCOBOL) 3.2.0 and run on Alpine Linux 3.24.3 (x86_64) in a Docker container.

Article author: Arthur Isaev

Banner image