mvn test and the surefire report

mvn test runs the default lifecycle up to the phase test: the code is compiled, the test code is compiled, and surefire runs the tests it finds with the JUnit 5 engine. It does not write a jar, and mvn package runs the same tests before it does. This article follows one green run through its log and into the two files it leaves, then runs verify to see what the next phase adds.

The project

Three tests over two methods, with a test dependency on the JUnit Jupiter aggregate, which is the only test library in the local repository of this image.

pom.xml

<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.example</groupId> <artifactId>calc</artifactId> <version>1.0.0</version> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <maven.compiler.release>21</maven.compiler.release> <junit.version>5.14.3</junit.version> </properties> <dependencies> <dependency> <groupId>org.junit.jupiter</groupId> <artifactId>junit-jupiter</artifactId> <version>${junit.version}</version> <scope>test</scope> </dependency> </dependencies> </project>

src/main/java/com/example/calc/Calc.java

package com.example.calc; public class Calc { public static int add(int a, int b) { return a + b; } public static int divide(int a, int b) { if (b == 0) { throw new IllegalArgumentException("division by zero"); } return a / b; } }

src/test/java/com/example/calc/CalcTest.java

package com.example.calc; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertThrows; import org.junit.jupiter.api.Test; class CalcTest { @Test void addsTwoNumbers() { assertEquals(4, Calc.add(1, 3)); } @Test void dividesTwoNumbers() { assertEquals(3, Calc.divide(9, 3)); } @Test void rejectsDivisionByZero() { IllegalArgumentException e = assertThrows(IllegalArgumentException.class, () -> Calc.divide(1, 0)); assertEquals("division by zero", e.getMessage()); } }

mvn test

mvn -B test > build.log 2>&1; echo "status: $?"; sed "s#$PWD#<project>#g" build.log

status: 0
[INFO] Scanning for projects...
[INFO]
[INFO] --------------------------< com.example:calc >--------------------------
[INFO] Building calc 1.0.0
[INFO] from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- resources:3.4.0:resources (default-resources) @ calc ---
[INFO] skip non existing resourceDirectory <project>/src/main/resources
[INFO]
[INFO] --- compiler:3.15.0:compile (default-compile) @ calc ---
[INFO] Recompiling the module because of <REASON>.
[INFO] Compiling 1 source file with javac [debug release 21] to target/classes
[INFO]
[INFO] --- resources:3.4.0:testResources (default-testResources) @ calc ---
[INFO] skip non existing resourceDirectory <project>/src/test/resources
[INFO]
[INFO] --- compiler:3.15.0:testCompile (default-testCompile) @ calc ---
[INFO] Recompiling the module because of <REASON>.
[INFO] Compiling 1 source file with javac [debug release 21] to target/test-classes
[INFO]
[INFO] --- surefire:3.5.4:test (default-test) @ calc ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.example.calc.CalcTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: <TIME> s -- in com.example.calc.CalcTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: <TIME>
[INFO] Finished at: <TIMESTAMP>
[INFO] ------------------------------------------------------------------------

Four goals ran before the count: resources, compile, testResources and testCompile, and the fifth is surefire. Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider says which engine surefire chose: the JUnit Platform, because the test dependency brings the Jupiter engine and no other provider is on the class path.

Running com.example.calc.CalcTest is one line per test class, and the counts after it are the four numbers of that class. The Results: block repeats them for the whole run, which is the line to read in a project with twenty test classes. Skipped: 0 counts the tests a disabled annotation or an assumption took out of the run.

What surefire found is decided by its defaults: the classes whose name ends with Test, Tests or TestCase in target/test-classes. A test class named CalcChecks is compiled and not run, and -Dtest is the switch that names one anyway; the surefire article shows both.

The reports

The run leaves a report per test class, in two formats:

ls -1 target/surefire-reports/*.txt target/surefire-reports/*.xml

target/surefire-reports/TEST-com.example.calc.CalcTest.xml
target/surefire-reports/com.example.calc.CalcTest.txt

The text file is the report the log is built from, and the XML file is the one a CI server reads. Both are named after the test class; the XML one has the TEST- prefix that Jenkins, GitLab and the others look for. ls in that folder shows more than these two files: surefire writes two .bin dumps per run whose names carry a timestamp, and a directory listing of them changes on every build.

cat target/surefire-reports/com.example.calc.CalcTest.txt

-------------------------------------------------------------------------------
Test set: com.example.calc.CalcTest
-------------------------------------------------------------------------------
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: <TIME> s -- in com.example.calc.CalcTest

Four lines: two rules, the name of the test class and the counts with the time. A failure adds the assertion message and the stack trace under it, and the surefire article shows that file for a red run. The XML file holds the same in the format the CI server parses: <testsuite tests="3" failures="0" ...> with one <testcase> per test.

test against verify

test is the fifteenth phase of the default lifecycle and the eighteenth, verify, is the last one before install. Between them the build runs the jar plugin and any goal a pom binds to the integration-test phases. This project binds none, so the only difference is the jar:

mvn -B verify > verify.log 2>&1; echo "status: $?"; sed "s#$PWD#<project>#g" verify.log

status: 0
[INFO] Scanning for projects...
[INFO]
[INFO] --------------------------< com.example:calc >--------------------------
[INFO] Building calc 1.0.0
[INFO] from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- resources:3.4.0:resources (default-resources) @ calc ---
[INFO] skip non existing resourceDirectory <project>/src/main/resources
[INFO]
[INFO] --- compiler:3.15.0:compile (default-compile) @ calc ---
[INFO] Nothing to compile - all classes are up to date.
[INFO]
[INFO] --- resources:3.4.0:testResources (default-testResources) @ calc ---
[INFO] skip non existing resourceDirectory <project>/src/test/resources
[INFO]
[INFO] --- compiler:3.15.0:testCompile (default-testCompile) @ calc ---
[INFO] Nothing to compile - all classes are up to date.
[INFO]
[INFO] --- surefire:3.5.4:test (default-test) @ calc ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.example.calc.CalcTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: <TIME> s -- in com.example.calc.CalcTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- jar:3.5.0:jar (default-jar) @ calc ---
[INFO] Building jar: <project>/target/calc-1.0.0.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: <TIME>
[INFO] Finished at: <TIMESTAMP>
[INFO] ------------------------------------------------------------------------

The excerpt is the end of the log: the tests ran again, unchanged, and the jar step came after them. On a project with integration tests, verify is the command a CI job runs, because it is the first phase that has seen everything the project can check. mvn test stays the command of the inner loop, because it stops before the jar.

Where to go next

The package article takes the jar this verify wrote and runs it, with and without a main class in the manifest. The surefire article goes into a red run, -Dtest and the two ways of skipping tests.

Article author: Arthur Isaev