A red test
A test that never fails proves nothing, so the first skill is reading a failure. This lesson writes a test with one wrong expected value, runs it, and reads every line the launcher prints: the tree, the failure list, the stack trace and the counts. Then it changes one character and the test is green.
The test that should fail
Calc.java adds two numbers correctly. The test asks for 5, but the method returns 4 for 1 and 3, so the assertion is false:
public class Calc {
static int add(int a, int b) {
return a + b;
}
}
import static org.junit.jupiter.api.Assertions.assertEquals;
import org.junit.jupiter.api.Test;
public class CalcTest {
@Test
void addsOneAndThree() {
assertEquals(5, Calc.add(1, 3));
}
}
The red run
The build succeeds: a wrong expected value is valid Java. The launcher is what reports the problem:
javac -Xlint:all -cp /opt/java-tools/junit-platform-console-standalone.jar -d out Calc.java CalcTest.java
junit execute --class-path out --scan-classpath --disable-ansi-colors
Thanks for using JUnit! Support its development at https://junit.org/sponsoring
╷
├─ JUnit Platform Suite ✔
├─ JUnit Jupiter ✔
│ └─ CalcTest ✔
│ └─ addsOneAndThree() ✘ expected: <5> but was: <4>
└─ JUnit Vintage ✔
Failures (1):
JUnit Jupiter:CalcTest:addsOneAndThree()
MethodSource [className = 'CalcTest', methodName = 'addsOneAndThree', methodParameterTypes = '']
=> org.opentest4j.AssertionFailedError: expected: <5> but was: <4>
org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)
org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:145)
org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:531)
CalcTest.addsOneAndThree(CalcTest.java:8)
java.base/java.lang.reflect.Method.invoke(Method.java:580)
java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
Test run finished after 757 ms
[ 4 containers found ]
[ 0 containers skipped ]
[ 4 containers started ]
[ 0 containers aborted ]
[ 4 containers successful ]
[ 0 containers failed ]
[ 1 tests found ]
[ 0 tests skipped ]
[ 1 tests started ]
[ 0 tests aborted ]
[ 0 tests successful ]
[ 1 tests failed ]
Reading the failure
The tree line for the method carries the first sentence of the failure: expected: <5> but was: <4>. The class line still has a check mark, because the class itself did not fail to run; the method did. The counts table says what the tree shows: one test found, started, and failed, and no test successful.
The Failures (1) section repeats the test in full. The first line names the engine, the class and the method: JUnit Jupiter:CalcTest:addsOneAndThree(). The MethodSource line is the machine-readable form a report file uses, with the class name, the method name and the parameter types, empty for a test without parameters.
The line that starts with => is the exception that the assertion threw: org.opentest4j.AssertionFailedError with the same message. The indented lines below it are the stack trace. Read it from the bottom up: the launcher called the method through reflection, and the first frame that belongs to your code is CalcTest.addsOneAndThree(CalcTest.java:8). That is where the assertion is, and the number is the line in the test file. The frames above are the JUnit classes that built the message.
A failure is not an error. An error is an exception the code did not expect, such as a NullPointerException; a failure is a test that ran and reported a wrong result. The tree prints both with a cross, and the counts table counts both as failed, but the failure list names the exception, which tells the two apart.
The exit status
The command returned 1. A script that runs the launcher and checks $? stops at a red run, which is what a build server does before it packages anything. The report is for the person who has to fix the test; the status is for the machine that must not go on.
The one-word fix
The method is right and the test is wrong, so change the expected value from 5 to 4: assertEquals(4, Calc.add(1, 3)). Build and run again, and the same report has a check mark on the method, no failure list, one test successful and status 0. The next lesson does exactly that with a class that has a real bug, so the fix is in the code and not in the test.
Exercises
- Change the expected value to 4, run the launcher, and confirm that the failure list is gone and the status is 0.
- Change the test so that it calls Calc.add(1, 3) twice and compares the two results. Does the failure message change?
- Add a second test that fails, and read the Failures (2) section. Which lines move?
- Run the red test with --select-method CalcTest#addsOneAndThree. Does the report name the method differently?
Questions
A new set of questions in a new order every time. Choose one answer for each, then check.
-
What does the tree line for a failed test add after the method name?
Answer
The first line of the failure message
The tree shows the method with a cross and the first sentence of the assertion message.
-
Where in the stack trace is the line of the test that called the assertion?
Answer
The first frame that names your test class
The frames above belong to JUnit; the first frame with your class and file is the test line.
-
What exit status does a red run return?
Answer
1
The launcher returns 1 when at least one test failed, whatever the number of failures.
-
What does this program print?
public class Q1 { public static void main(String[] args) { int a = 5; int b = 3; System.out.println(a - b); } }Answer
2
The subtraction gives 2, and println prints it.
-
What does this program print?
public class Q2 { public static void main(String[] args) { String text = "red"; System.out.println(text.equals("red")); } }Answer
true
equals compares the contents and returns a boolean, which println prints as true.
-
What does this program print?
public class Q3 { public static void main(String[] args) { int total = 0; for (int n = 1; n != 5; n++) { total += 2; } System.out.println(total); } }Answer
8
The loop runs four times and adds 2 each time, so the total is 8.
-
What does this program print?
public class Q4 { public static void main(String[] args) { System.out.println(3 + 4 + "!"); } }Answer
7!
The two numbers are added before the string is joined, so the result is 7!.
-
A test expects 5 and the method returns 4. Which exception does
assertEqualsthrow?Answer
org.opentest4j.AssertionFailedError
JUnit 5 builds its failure as an AssertionFailedError from the opentest4j library.
-
What is the difference between a failure and an error in the report?
Answer
A failure is a wrong result; an error is an unexpected exception
Both count as failed, but a failure comes from an assertion and an error from any other exception.
-
What does this program print?
public class Q5 { public static void main(String[] args) { int[] values = {2, 4, 6}; System.out.println(values[1]); } }Answer
4
Arrays are indexed from 0, so values[1] is the second element, 4.
The code on this page was built with OpenJDK 21.0.13 (javac) and run with JUnit Platform Console Launcher 1.14.3 on Alpine Linux in a Docker container.
Article author: Arthur Isaev