Bytes and binary files

A text file is bytes plus an encoding. A binary file is only the bytes, and the program decides what each of them means. Java's tool for that is ByteBuffer: a block of bytes with a position, methods that write an int or a double into it, and the same methods to read them back.

A binary record in a ByteBuffer

The record below is an id, a name and a price. The format has no length prefix for the string but writes the number of bytes first, which is how a reader knows where the name ends and the price begins:

import java.nio.ByteBuffer; import java.util.HexFormat; public class Record { record Item(int id, String name, double price) { } public static void main(String[] args) { Item item = new Item(42, "widget", 12.5); byte[] name = item.name().getBytes(java.nio.charset.StandardCharsets.UTF_8); ByteBuffer buffer = ByteBuffer.allocate(4 + 4 + name.length + 8); buffer.putInt(item.id()); buffer.putInt(name.length); buffer.put(name); buffer.putDouble(item.price()); byte[] bytes = buffer.array(); System.out.println("bytes: " + bytes.length); System.out.println("hex: " + HexFormat.of().formatHex(bytes)); ByteBuffer read = ByteBuffer.wrap(bytes); int id = read.getInt(); int length = read.getInt(); byte[] text = new byte[length]; read.get(text); double price = read.getDouble(); Item back = new Item(id, new String(text, java.nio.charset.StandardCharsets.UTF_8), price); System.out.println(back); System.out.println("same: " + back.equals(item)); System.out.println("remaining: " + read.remaining()); } }

javac -Xlint:all Record.java
java Record

bytes: 22
hex: 0000002a000000067769646765744029000000000000
Item[id=42, name=widget, price=12.5]
same: true
remaining: 0

The hex line is the format read byte by byte. 0000002a is 42 in four bytes, 00000006 is the length of the name, 776964676574 is the six bytes of widget and 4029000000000000 is 12.5 as an IEEE double. The last line proves that the reader consumed exactly the bytes the writer wrote, which is the check to make when a format is new.

readAllBytes and write

A file of bytes needs no buffer unless the program is parsing it. Files.write takes a byte array and Files.readAllBytes returns one; the program prints the bytes in hex and sums them as unsigned values:

import java.io.IOException; import java.nio.charset.StandardCharsets; import java.nio.file.Files; import java.nio.file.Path; import java.util.HexFormat; public class File { public static void main(String[] args) throws IOException { Path file = Path.of("data.bin"); byte[] bytes = {0x01, 0x02, 0x7f, (byte) 0x80, (byte) 0xff}; Files.write(file, bytes); System.out.println("written: " + Files.size(file)); byte[] back = Files.readAllBytes(file); System.out.println("read: " + back.length); System.out.println("hex: " + HexFormat.of().formatHex(back)); int sum = 0; for (byte b : back) { sum += b & 0xff; } System.out.println("unsigned sum: " + sum); Files.delete(file); Path text = Path.of("text.bin"); Files.write(text, "ab".getBytes(StandardCharsets.UTF_8)); System.out.println("text bytes: " + java.util.Arrays.toString(Files.readAllBytes(text))); Files.delete(text); } }

javac -Xlint:all File.java
java File

written: 5
read: 5
hex: 01027f80ff
unsigned sum: 513
text bytes: [97, 98]

The literal (byte) 0x80 is a cast because a byte is signed and 0x80 does not fit; the mask

Byte order

An int is four bytes, and the order of the four is a decision of the format. Java writes big endian by default, which is the order of the network; a file from a little-endian machine reads as a different number unless the buffer is told:

import java.nio.ByteBuffer; import java.nio.ByteOrder; import java.util.HexFormat; public class Order { public static void main(String[] args) { ByteBuffer big = ByteBuffer.allocate(4).order(ByteOrder.BIG_ENDIAN); big.putInt(0x01020304); ByteBuffer little = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN); little.putInt(0x01020304); HexFormat hex = HexFormat.of(); System.out.println("big: " + hex.formatHex(big.array())); System.out.println("little: " + hex.formatHex(little.array())); ByteBuffer read = ByteBuffer.wrap(big.array()).order(ByteOrder.BIG_ENDIAN); System.out.println("back: " + Integer.toHexString(read.getInt())); ByteBuffer wrong = ByteBuffer.wrap(big.array()).order(ByteOrder.LITTLE_ENDIAN); System.out.println("wrong: " + Integer.toHexString(wrong.getInt())); } }

javac -Xlint:all Order.java
java Order

big: 01020304
little: 04030201
back: 1020304
wrong: 4030201

The two writes store the same number in the opposite byte order, and the last two lines read one of them the right way and the wrong way: 1020304 is the value, 4030201 is what the same bytes mean when they are read in the other order. A format that leaves the order unsaid is a format that a second program will read wrongly, and the JDK's DataOutputStream and DataInputStream are the older pair that fixed it to big endian.

Rules for binary data

Decide the order and write it down, because a number has no order of its own. Prefix a variable-length field with its length, as the record does, or delimit it with a byte that cannot appear in the data. Fix the size of every field in the format instead of using whatever the platform gives, which is why a file format uses int and not long for a count when four bytes are enough. And use readAllBytes only for a file that fits in memory: a binary file can be read through a FileChannel with a buffer of a few kilobytes, which is the subject of lesson 124 from the memory side.

A binary format that has to be read by another language is where these rules stop being style: a C program writes a struct in the order of the machine and pads its fields, and the Java reader must be told both. The format documents them, or the two programs disagree on byte three and nobody knows why.

Exercises

  1. Add a boolean field to the record as one byte and print the new length.
  2. Write the same bytes to a file and read them back into a ByteBuffer.
  3. Write the number 1 as a little-endian int and as a big-endian int and compare the hex.
  4. Read the price of the record with the wrong byte order and explain the number.

Questions

A new set of questions in a new order every time. Choose one answer for each, then check.

  1. What does ByteBuffer.putInt write?

    Answer

    Four bytes of the int, in the order the buffer is set to

    The buffer writes the machine form of the value in four bytes, and the order is a property of it.

  2. Why does a binary record store the length of a name?

    Answer

    So the reader knows where the name ends and the next field begins

    A variable-length field needs a way to find its end, and the length is the one that survives any content.

  3. What is the default byte order of a ByteBuffer?

    Answer

    Big endian

    Java writes big endian by default, which is also the network order.

  4. What does this program print? import java.nio.ByteBuffer; import java.util.HexFormat; public class Q1 { public static void main(String[] args) { ByteBuffer b = ByteBuffer.allocate(8); b.putInt(1); b.putInt(256); System.out.println(HexFormat.of().formatHex(b.array())); } }

    Answer

    0000000100000100

    The first int is four bytes of mostly zeros and the second is 0x100.

  5. Why is the mask b & 0xff needed when a byte is summed?

    Answer

    A byte is signed, so 0xff would add -1 instead of 255

    The widening of a negative byte keeps the sign, and the mask asks for the unsigned value.

  6. What does buffer.remaining() answer after a full read?

    Answer

    Zero, because the position reached the limit

    The position moves with each get, and remaining is the distance to the limit.

  7. Which pair of classes writes binary data in big endian with a fixed size per type?

    Answer

    DataOutputStream and DataInputStream

    The data streams fix the order and the size, which is what a portable format needs.

  8. What does this program print? import java.nio.ByteBuffer; public class Q2 { public static void main(String[] args) { ByteBuffer b = ByteBuffer.allocate(4); b.putInt(0x00000001); System.out.println(b.array()[3]); System.out.println(b.array()[0]); } }

    Answer

    1 / 0

    Big endian puts the low byte last, so index three is one and index zero is zero.

  9. What does this program print? import java.nio.ByteBuffer; import java.util.HexFormat; public class Q3 { public static void main(String[] args) { byte[] bytes = {1, 2, 3}; ByteBuffer b = ByteBuffer.wrap(bytes); System.out.println(b.get()); System.out.println(b.remaining()); System.out.println(HexFormat.of().formatHex(bytes)); } }

    Answer

    1 / 2 / 010203

    The get returns the first byte and moves the position, leaving two bytes.

  10. Why should a binary format fix the size of every field?

    Answer

    So the positions of the later fields are known to every reader

    A reader that does not know a field size cannot find the next one.

The code on this page was built with OpenJDK 21.0.13 (javac and java) and run on Alpine Linux in a Docker container.

Article author: Arthur Isaev