Assertions and a failure report
A test is only as good as its comparison. JUnit 5 puts every comparison in a static method of Assertions, each one named after what it checks, and a failed one throws an exception with the expected value, the actual value and any message the test gave. This page goes through the family and then runs a test that fails, because the failure report is the part a person reads every day.
The class under test
Five methods over an array of numbers, and one that answers null for a negative value so that a test has something to check with assertNull. The main prints all five results, which is the program side of the file:
import java.util.Arrays;
public class Stats {
static int sum(int[] values) {
int total = 0;
for (int value : values) {
total += value;
}
return total;
}
static double mean(int[] values) {
if (values.length == 0) {
throw new IllegalArgumentException("no values");
}
return (double) sum(values) / values.length;
}
static int max(int[] values) {
return Arrays.stream(values).max().orElseThrow(() -> new IllegalArgumentException("no values"));
}
static String label(int value) {
return value < 0 ? null : "value " + value;
}
public static void main(String[] args) {
int[] values = {4, 8, 15, 16, 23, 42};
System.out.println("sum: " + sum(values));
System.out.println("mean: " + mean(values));
System.out.println("max: " + max(values));
System.out.println("label: " + label(7));
System.out.println("label of -1: " + label(-1));
}
}
The six numbers sum to 108 and their mean is 18.0 because 108 divided by 6 is exact; the maximum is 42 and the two labels show the null. Those are the values the tests assert, which means a reader can check the expected numbers by hand before running anything.
The assertions
Six tests, one per kind of comparison. The expected value comes first in every method and the actual value second, which is the order the report uses and the order a failure reads back. assertArrayEquals compares element by element, because an array does not override equals and assertEquals on two arrays would compare references:
import static org.junit.jupiter.api.Assertions.assertArrayEquals;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;
@TestMethodOrder(MethodOrderer.MethodName.class)
public class StatsTest {
@Test
void addsTheValues() {
assertEquals(108, Stats.sum(new int[] {4, 8, 15, 16, 23, 42}));
}
@Test
void comparesDoublesWithADelta() {
assertEquals(18.0, Stats.mean(new int[] {4, 8, 15, 16, 23, 42}), 0.0001);
}
@Test
void findsTheMaximum() {
assertEquals(42, Stats.max(new int[] {4, 8, 15, 16, 23, 42}));
}
@Test
void labelsAPositiveValue() {
assertNotNull(Stats.label(7));
assertTrue(Stats.label(7).startsWith("value"));
}
@Test
void hasNoLabelForANegative() {
assertNull(Stats.label(-1), "a negative value has no label");
}
@Test
void seesTheSameArray() {
int[] sorted = {4, 8, 15};
assertArrayEquals(new int[] {4, 8, 15}, sorted);
assertFalse(sorted.length == 0);
}
}
Two methods hold more than one assertion, and that is a decision a test makes: the first failure ends the method, so the second assertion of labelsAPositiveValue runs only if the first held. A test that must report every failure at once uses assertAll, which collects the failures and throws one exception that lists them, and the exceptions page shows it.
Doubles and a delta
There is no assertEquals(double, double) without a third argument in JUnit 5, and the reason is the arithmetic: 0.1 + 0.2 is not 0.3 in binary floating point, so an exact comparison of two computed doubles fails for reasons a reader cannot see. The third argument is the tolerance, and the assertion holds when the distance between the two values is at most it:
assertEquals(18.0, Stats.mean(...), 0.0001) accepts anything from 17.9999 to 18.0001. The tolerance is a decision of the test: too wide and a real error passes, too narrow and the test is flaky. For money the course uses BigDecimal and compares exactly, which is the other way to avoid the question.
The green run
The two classes are compiled together with the launcher jar on the class path, and the program is run first, because a class that is wrong as a program is usually wrong as a subject too:
javac -Xlint:all -cp /opt/java-tools/junit-platform-console-standalone.jar -d out Stats.java StatsTest.java
java -cp out Stats
sum: 108
mean: 18.0
max: 42
label: value 7
label of -1: null
The launcher then runs the six tests of StatsTest. The command selects the class by name instead of scanning the folder, because the folder will hold a second test class in a moment:
junit execute --class-path out --select-class StatsTest --disable-ansi-colors
Thanks for using JUnit! Support its development at https://junit.org/sponsoring
╷
├─ JUnit Platform Suite ✔
├─ JUnit Jupiter ✔
│ └─ StatsTest ✔
│ ├─ addsTheValues() ✔
│ ├─ comparesDoublesWithADelta() ✔
│ ├─ findsTheMaximum() ✔
│ ├─ hasNoLabelForANegative() ✔
│ ├─ labelsAPositiveValue() ✔
│ └─ seesTheSameArray() ✔
└─ JUnit Vintage ✔
Test run finished after 214 ms
[ 4 containers found ]
[ 0 containers skipped ]
[ 4 containers started ]
[ 0 containers aborted ]
[ 4 containers successful ]
[ 0 containers failed ]
[ 6 tests found ]
[ 0 tests skipped ]
[ 6 tests started ]
[ 0 tests aborted ]
[ 6 tests successful ]
[ 0 tests failed ]
The message of a failure
The last argument of an assertion is a message, and it is worth writing whenever the expected value does not explain itself. The test below fails on purpose: it expects the sum of 1 and 2 to be 5, which is wrong, and it passes the sentence the sum of 1 and 2 as the message. The class is compiled on its own, with the compiled folder on the class path so it can see Stats:
import static org.junit.jupiter.api.Assertions.assertEquals;
import org.junit.jupiter.api.Test;
public class BrokenTest {
@Test
void expectsTheWrongSum() {
assertEquals(5, Stats.sum(new int[] {1, 2}), "the sum of 1 and 2");
}
}
javac -Xlint:all -cp /opt/java-tools/junit-platform-console-standalone.jar:out -d out BrokenTest.java
junit execute --class-path out --select-class BrokenTest --disable-ansi-colors
echo "status: $?"
Thanks for using JUnit! Support its development at https://junit.org/sponsoring
╷
├─ JUnit Platform Suite ✔
├─ JUnit Jupiter ✔
│ └─ BrokenTest ✔
│ └─ expectsTheWrongSum() ✘ the sum of 1 and 2 ==> expected: <5> but was: <3>
└─ JUnit Vintage ✔
Failures (1):
JUnit Jupiter:BrokenTest:expectsTheWrongSum()
MethodSource [className = 'BrokenTest', methodName = 'expectsTheWrongSum', methodParameterTypes = '']
=> org.opentest4j.AssertionFailedError: the sum of 1 and 2 ==> expected: <5> but was: <3>
org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)
org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:563)
BrokenTest.expectsTheWrongSum(BrokenTest.java:8)
java.base/java.lang.reflect.Method.invoke(Method.java:580)
java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
Test run finished after 319 ms
[ 4 containers found ]
[ 0 containers skipped ]
[ 4 containers started ]
[ 0 containers aborted ]
[ 4 containers successful ]
[ 0 containers failed ]
[ 1 tests found ]
[ 0 tests skipped ]
[ 1 tests started ]
[ 0 tests aborted ]
[ 0 tests successful ]
[ 1 tests failed ]
status: 1
A red run
Read the third line of the tree against the counts. The class is marked ✔, because the class itself was discovered and run; the method is marked ✘ and carries the message and the two values, which is the sentence the test wrote followed by expected: <5> but was: <3>. The failure block repeats it with the full stack trace, and the last count is one, not zero.
The exit status of the launcher is 1, which the last line of the command block proves: the echo after the run reads $? and prints it. A build server does not read the tree; it reads that status. A green run exits 0, and that is the whole contract between a test runner and the tool that called it.
Reading the counts
Six tests found, six successful in the green run above; one found and one failed in the red one. The line that matters on a daily basis is the failed count, and the second is the aborted count: an aborted test is one that threw something the framework does not treat as an assertion failure, such as an exception from the code under test, and the two are counted apart on purpose.
A skipped test is a third state, written with @Disabled or with an assumption. A project that keeps a test disabled for a year has a test nobody trusts, and the report keeps showing it, which is the point: the count of skipped tests is a debt a reader can see.
What comes next
Six tests with six inputs are six methods, and a seventh input means a seventh method. The next page replaces that repetition with a parameterised test: one method, a source of cases, and a report that names the case that failed.
The Java course covers the same assertions in lesson 127, and the design of the exception a failed assertion throws is part of the exception design lesson.
The code on this page was compiled with OpenJDK 21.0.13 (javac) and run with the JUnit Platform Console Launcher 1.14.3 on Alpine Linux in a Docker container. The time in each report is that machine's and changes between runs.
Article author: Arthur Isaev