Surefire, JUnit 5 and the reports

Surefire is the plugin of the test phase. Its job is to find the test classes, run them with the JUnit Platform, and write a report per class in two formats. A green run takes a page of the test phase article; this one is about the other case, because a red run is where surefire has something to say.

The project

Two tests that pass and one that does not: the first assertion expects 5 where the method returns 4.

pom.xml

<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.example</groupId> <artifactId>calc</artifactId> <version>1.0.0</version> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <maven.compiler.release>21</maven.compiler.release> <junit.version>5.14.3</junit.version> </properties> <dependencies> <dependency> <groupId>org.junit.jupiter</groupId> <artifactId>junit-jupiter</artifactId> <version>${junit.version}</version> <scope>test</scope> </dependency> </dependencies> </project>

src/main/java/com/example/calc/Calc.java

package com.example.calc; public class Calc { public static int add(int a, int b) { return a + b; } public static int times(int a, int b) { return a * b; } }

src/test/java/com/example/calc/CalcTest.java

package com.example.calc; import static org.junit.jupiter.api.Assertions.assertEquals; import org.junit.jupiter.api.Test; class CalcTest { @Test void addsTwoNumbers() { assertEquals(5, Calc.add(1, 3)); } @Test void multipliesTwoNumbers() { assertEquals(12, Calc.times(3, 4)); } }

A red run

mvn -B test > build.log 2>&1; echo "status: $?"; sed "s#$PWD#<project>#g" build.log

status: 1
[INFO] Scanning for projects...
[INFO]
[INFO] --------------------------< com.example:calc >--------------------------
[INFO] Building calc 1.0.0
[INFO] from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- resources:3.4.0:resources (default-resources) @ calc ---
[INFO] skip non existing resourceDirectory <project>/src/main/resources
[INFO]
[INFO] --- compiler:3.15.0:compile (default-compile) @ calc ---
[INFO] Recompiling the module because of <REASON>.
[INFO] Compiling 1 source file with javac [debug release 21] to target/classes
[INFO]
[INFO] --- resources:3.4.0:testResources (default-testResources) @ calc ---
[INFO] skip non existing resourceDirectory <project>/src/test/resources
[INFO]
[INFO] --- compiler:3.15.0:testCompile (default-testCompile) @ calc ---
[INFO] Recompiling the module because of <REASON>.
[INFO] Compiling 1 source file with javac [debug release 21] to target/test-classes
[INFO]
[INFO] --- surefire:3.5.4:test (default-test) @ calc ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.example.calc.CalcTest
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: <TIME> s <<< FAILURE! -- in com.example.calc.CalcTest
[ERROR] com.example.calc.CalcTest.addsTwoNumbers -- Time elapsed: <TIME> s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <5> but was: <4>
at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)
at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:145)
at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:531)
at com.example.calc.CalcTest.addsTwoNumbers(CalcTest.java:10)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] CalcTest.addsTwoNumbers:10 expected: <5> but was: <4>
[INFO]
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: <TIME>
[INFO] Finished at: <TIMESTAMP>
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.5.4:test (default-test) on project calc: There are test failures.
[ERROR]
[ERROR] See <project>/target/surefire-reports for the individual test results.
[ERROR] See dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

The block reads from the inside out. Tests run: 2, Failures: 1 is the count of the class; the line under it names the failing method with its time and the <<< FAILURE! marker. The assertion message, expected: <5> but was: <4>, is the one the JUnit 5 assertion throws, and the stack trace under it starts at the assertion library and ends in the test method with its line number.

Then Results: lists the failures as CalcTest.addsTwoNumbers:8 expected: <5> but was: <4>, which is the line a CI job shows and the file and line to open. The build fails at the surefire goal with There are test failures., and the two See ... lines point at the reports and the dump files.

The report of the failure

The text file of the test class is the log of the class without the Maven lines:

cat target/surefire-reports/com.example.calc.CalcTest.txt

-------------------------------------------------------------------------------
Test set: com.example.calc.CalcTest
-------------------------------------------------------------------------------
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: <TIME> s <<< FAILURE! -- in com.example.calc.CalcTest
com.example.calc.CalcTest.addsTwoNumbers -- Time elapsed: <TIME> s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <5> but was: <4>
at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)
at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:145)
at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:531)
at com.example.calc.CalcTest.addsTwoNumbers(CalcTest.java:10)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)

The XML file next to it, TEST-com.example.calc.CalcTest.xml, holds the same failure for a CI server: a <testsuite> with the counts, one <testcase> per test, and a <failure> element with the message and the stack trace. It also records forty-odd properties of the run, the class path and the JDK among them, which is why the file is machine-readable rather than something to read.

One test by name

-Dtest selects tests by name: a class, a method after #, a wildcard, or a list separated by commas. The failing method alone:

mvn -B test -Dtest=CalcTest#addsTwoNumbers > one.log 2>&1; echo "status: $?"; sed "s#$PWD#<project>#g" one.log

status: 1
[INFO] Scanning for projects...
[INFO]
[INFO] --------------------------< com.example:calc >--------------------------
[INFO] Building calc 1.0.0
[INFO] from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.example.calc.CalcTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: <TIME> s <<< FAILURE! -- in com.example.calc.CalcTest
[ERROR] com.example.calc.CalcTest.addsTwoNumbers -- Time elapsed: <TIME> s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <5> but was: <4>
at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)
at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:145)
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

One test, one failure, the same message. A method that is not in the class, or a class that does not match, is not an error by default: the build succeeds with No tests to run. and the exit status 0. A CI job that filters tests usually adds -Dsurefire.failIfNoSpecifiedTests=false when the filter may be empty, and leaves it out when an empty filter should fail.

Which classes run

Surefire runs the classes under target/test-classes whose name matches its includes: **/Test*.java, **/*Test.java, **/*Tests.java and **/*TestCase.java. The pattern is about the file name, not the annotations, so a class called CalcChecks with two @Test methods is compiled and not run.

A project that needs other names configures <includes> and <excludes> in the plugin, and a project that needs a filter changes it on the command line with -Dtest. The list of the tests that ran is always in the log and in the counts, never in the annotations.

Two failures in one class do not stop each other: surefire runs every test of the class and reports all of them, which is why Failures: 2 is a normal state for a red run and why a project reads the whole report before fixing one thing. The JUnit 5 course goes through the assertions themselves, and its lesson on the report reads the launcher's own output, which is the same tree surefire prints.

Where to go next

The parent pom article starts the part of the course about projects with more than one module, where the plugin versions are written once for all of them.

Article author: Arthur Isaev