Bytes and binary files
A text file is bytes plus an encoding. A binary file is only the bytes, and the program decides what each of them means. Java's tool for that is ByteBuffer: a block of bytes with a position, methods that write an int or a double into it, and the same methods to read them back.
A binary record in a ByteBuffer
The record below is an id, a name and a price. The format has no length prefix for the string but writes the number of bytes first, which is how a reader knows where the name ends and the price begins:
import java.nio.ByteBuffer;
import java.util.HexFormat;
public class Record {
record Item(int id, String name, double price) {
}
public static void main(String[] args) {
Item item = new Item(42, "widget", 12.5);
byte[] name = item.name().getBytes(java.nio.charset.StandardCharsets.UTF_8);
ByteBuffer buffer = ByteBuffer.allocate(4 + 4 + name.length + 8);
buffer.putInt(item.id());
buffer.putInt(name.length);
buffer.put(name);
buffer.putDouble(item.price());
byte[] bytes = buffer.array();
System.out.println("bytes: " + bytes.length);
System.out.println("hex: " + HexFormat.of().formatHex(bytes));
ByteBuffer read = ByteBuffer.wrap(bytes);
int id = read.getInt();
int length = read.getInt();
byte[] text = new byte[length];
read.get(text);
double price = read.getDouble();
Item back = new Item(id, new String(text, java.nio.charset.StandardCharsets.UTF_8), price);
System.out.println(back);
System.out.println("same: " + back.equals(item));
System.out.println("remaining: " + read.remaining());
}
}
javac -Xlint:all Record.java
java Record
bytes: 22
hex: 0000002a000000067769646765744029000000000000
Item[id=42, name=widget, price=12.5]
same: true
remaining: 0
The hex line is the format read byte by byte. 0000002a is 42 in four bytes, 00000006 is the length of the name, 776964676574 is the six bytes of widget and 4029000000000000 is 12.5 as an IEEE double. The last line proves that the reader consumed exactly the bytes the writer wrote, which is the check to make when a format is new.
readAllBytes and write
A file of bytes needs no buffer unless the program is parsing it. Files.write takes a byte array and Files.readAllBytes returns one; the program prints the bytes in hex and sums them as unsigned values:
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.HexFormat;
public class File {
public static void main(String[] args) throws IOException {
Path file = Path.of("data.bin");
byte[] bytes = {0x01, 0x02, 0x7f, (byte) 0x80, (byte) 0xff};
Files.write(file, bytes);
System.out.println("written: " + Files.size(file));
byte[] back = Files.readAllBytes(file);
System.out.println("read: " + back.length);
System.out.println("hex: " + HexFormat.of().formatHex(back));
int sum = 0;
for (byte b : back) {
sum += b & 0xff;
}
System.out.println("unsigned sum: " + sum);
Files.delete(file);
Path text = Path.of("text.bin");
Files.write(text, "ab".getBytes(StandardCharsets.UTF_8));
System.out.println("text bytes: " + java.util.Arrays.toString(Files.readAllBytes(text)));
Files.delete(text);
}
}
javac -Xlint:all File.java
java File
written: 5
read: 5
hex: 01027f80ff
unsigned sum: 513
text bytes: [97, 98]
The literal (byte) 0x80 is a cast because a byte is signed
and 0x80 does not fit; the mask
An int is four bytes, and the order of the four is a decision
of the format. Java writes big endian by default, which is the order of the network;
a file from a little-endian machine reads as a different number unless the buffer is
told:
javac -Xlint:all Order.java
big: 01020304
The two writes store the same number in the opposite byte order, and the last two
lines read one of them the right way and the wrong way: 1020304 is the value,
4030201 is what the same bytes mean when they are read in the other order. A format
that leaves the order unsaid is a format that a second program will read wrongly,
and the JDK's DataOutputStream and
DataInputStream are the older pair that fixed it to big
endian.
Decide the order and write it down, because a number has no order of its own. Prefix
a variable-length field with its length, as the record does, or delimit it with a
byte that cannot appear in the data. Fix the size of every field in the format
instead of using whatever the platform gives, which is why a file format uses
int and not long for a count when
four bytes are enough. And use readAllBytes only for a file
that fits in memory: a binary file can be read through a
FileChannel with a buffer of a few kilobytes, which is
the subject of lesson 124 from the memory side.
A binary format that has to be read by another language is where these rules stop
being style: a C program writes a struct in the order of the
machine and pads its fields, and the Java reader must be told both. The format
documents them, or the two programs disagree on byte three and nobody knows why.
A new set of questions in a new order every time. Choose one answer for each, then check. What does Four bytes of the int, in the order the buffer is set to The buffer writes the machine form of the value in four bytes, and the order is a property of it. Why does a binary record store the length of a name? So the reader knows where the name ends and the next field begins A variable-length field needs a way to find its end, and the length is the one that survives any content. What is the default byte order of a ByteBuffer? Big endian Java writes big endian by default, which is also the network order. What does this program print?
0000000100000100 The first int is four bytes of mostly zeros and the second is 0x100. Why is the mask A byte is signed, so 0xff would add -1 instead of 255 The widening of a negative byte keeps the sign, and the mask asks for the unsigned value. What does Zero, because the position reached the limit The position moves with each get, and remaining is the distance to the limit. Which pair of classes writes binary data in big endian with a fixed size per type? DataOutputStream and DataInputStream The data streams fix the order and the size, which is what a portable format needs. What does this program print?
1 / 0 Big endian puts the low byte last, so index three is one and index zero is zero. What does this program print?
1 / 2 / 010203 The get returns the first byte and moves the position, leaving two bytes. Why should a binary format fix the size of every field? So the positions of the later fields are known to every reader A reader that does not know a field size cannot find the next one.
The code on this page was built with OpenJDK 21.0.13 (javac
and java) and run on Alpine Linux in a Docker container.
Article author: Arthur Isaev
Byte order
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
import java.util.HexFormat;
public class Order {
public static void main(String[] args) {
ByteBuffer big = ByteBuffer.allocate(4).order(ByteOrder.BIG_ENDIAN);
big.putInt(0x01020304);
ByteBuffer little = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN);
little.putInt(0x01020304);
HexFormat hex = HexFormat.of();
System.out.println("big: " + hex.formatHex(big.array()));
System.out.println("little: " + hex.formatHex(little.array()));
ByteBuffer read = ByteBuffer.wrap(big.array()).order(ByteOrder.BIG_ENDIAN);
System.out.println("back: " + Integer.toHexString(read.getInt()));
ByteBuffer wrong = ByteBuffer.wrap(big.array()).order(ByteOrder.LITTLE_ENDIAN);
System.out.println("wrong: " + Integer.toHexString(wrong.getInt()));
}
}
java Order
little: 04030201
back: 1020304
wrong: 4030201
Rules for binary data
Exercises
Questions
ByteBuffer.putInt write?Answer
Answer
Answer
import java.nio.ByteBuffer;
import java.util.HexFormat;
public class Q1 {
public static void main(String[] args) {
ByteBuffer b = ByteBuffer.allocate(8);
b.putInt(1);
b.putInt(256);
System.out.println(HexFormat.of().formatHex(b.array()));
}
}Answer
b & 0xff needed when a byte is summed?Answer
buffer.remaining() answer after a full read?Answer
Answer
import java.nio.ByteBuffer;
public class Q2 {
public static void main(String[] args) {
ByteBuffer b = ByteBuffer.allocate(4);
b.putInt(0x00000001);
System.out.println(b.array()[3]);
System.out.println(b.array()[0]);
}
}Answer
import java.nio.ByteBuffer;
import java.util.HexFormat;
public class Q3 {
public static void main(String[] args) {
byte[] bytes = {1, 2, 3};
ByteBuffer b = ByteBuffer.wrap(bytes);
System.out.println(b.get());
System.out.println(b.remaining());
System.out.println(HexFormat.of().formatHex(bytes));
}
}Answer
Answer