Make the most of the logs when running Gradle tests

Both Maven and Gradle generate nicely formatted HTML reports. Moreover, Maven’s Surefire plugin outputs all essential information directly to the terminal. That provides an instant access to stack trace (in case of failures), logs from individual tests etc. Also, there is a nice summary at the end:

[bash]
——————————————————-
T E S T S
——————————————————-
Running com.mycompany.app.AppTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[/bash]

Gradle doesn’t provide such capabilities out of the box, but it doesn’t take too much effort to arrive at a similar result.

Let’s start with generating a sample project:

[bash]
$ mkdir gradle-sample-app && cd $_ && gradle init –type java-library
[/bash]

Now run the tests:

[bash]
$ gradle test
:compileJava
:processResources UP-TO-DATE
:classes
:compileTestJava
:processTestResources UP-TO-DATE
:testClasses
:test
BUILD SUCCESSFUL
[/bash]

As you can see there isn’t much information about the test suite. “Build successful”, that’s about it.

Imagine our sample app turns into a complex project with hundreds of unit tests. Surely it would be beneficial to see how many tests there are in total or how many have been skipped. Can something be done about it? Absolutely.

Open the build.gradle and add the following section:

[bash]
test {
     testLogging {
          events “started”, “passed”, “skipped”, “failed”, “standardOut”, “standardError”
     }
}
[/bash]

Rerun the test task and you should see the status of individual test methods.

[bash]
LibraryTest > testSomeLibraryMethod STARTED
LibraryTest > testSomeLibraryMethod PASSED
BUILD SUCCESSFUL
[/bash]

Let’s add a new test making use of the standard output:

[bash]
$ vi src/test/java/LibraryTest.java
[/bash]

[java]
..
@Test public void testAnotherLibraryMethod() {
  System.out.println(“test output”);
}
[/java]

Rerun the test task and enjoy watching the test logs being part of the build:

[bash]
$ gradle test
LibraryTest > testSomeLibraryMethod STARTED
LibraryTest > testSomeLibraryMethod PASSED
LibraryTest > testAnotherLibraryMethod STARTED
LibraryTest > testAnotherLibraryMethod STANDARD_OUT
test output
LibraryTest > testAnotherLibraryMethod PASSED
BUILD SUCCESSFUL
[/bash]

Time for a cleanup.

[code]
test {
     testLogging {
showStandardOutput = true
          events “started”, “passed”, “skipped”, “failed”<span style=”color: #000000;”><del>, “standardOut”, “standardError”</del> </span>
     }
}
[/code]

What if there is a failure? Let’s add a failing test.

[java]
@Test public void testSetToFail() {
    assertEquals(“foo”, “Foo”);
}
[/java]

The result is okay, but not tremendously helpful.

[bash]
<pre>LibraryTest > testSetToFail FAILED
org.junit.ComparisonFailure
at LibraryTest.java:20
[/bash]

Let’s make the build be more explicit about what exactly went wrong.

[code]
test {
     testLogging {
showStandardOutput = true
exceptionFormat = ‘full’
          events “started”, “passed”, “skipped”, “failed”
     }
}
[/code]

Now you can easily see what the root cause is.

[bash]
LibraryTest > testSetToFail FAILED
org.junit.ComparisonFailure: expected:<[f]oo> but was:<[F]oo>
at org.junit.Assert.assertEquals(Assert.java:115)
at org.junit.Assert.assertEquals(Assert.java:144)
at LibraryTest.testSetToFail(LibraryTest.java:20)
[/bash]

Finally, wouldn’t be nice to get a Maven-style summary at the of the build? Sure, just add the afterSuite section below (credit: Ori Peleg)

[code]
test {
     testLogging {
exceptionFormat = ‘full’
showStandardOutput = true
          events “started”, “passed”, “skipped”, “failed”
afterSuite { desc, result ->
// Match the outermost suite
if (!desc.parent) {
println “Tests run: ${result.testCount},
Failures: ${result.failedTestCount},
Skipped: ${result.skippedTestCount}”
}
}
     }
}
[/code]

This is what you get, hope it looks familiar.

[bash]
Tests run: 3, Failures: 1, Skipped: 0
[/bash]

Right, how about the skipped tests? Well, it’s not what you think it is. Suppose there is an @Ignor(ed) test.

[java]
@org.junit.Ignore public void ignoredTest() {
System.out.println(“That shouldn’t happen”);
}
[/java]

Now there are five tests in total, two of which are passing, one fails and one is ignored. Had we used Maven, the skipped test would have been counted.

[bash]
Tests run: 3, Failures: 1, Errors: 0, Skipped: 1
[/bash]

That’s not the case with Gradle (tested on version 2.6).  As you can see the skipped test literally doesn’t count.

[bash]
Tests run: 3, Failures: 1, Skipped: 0
[/bash]

Hope you enjoyed this brief post about Gradle’s logging tweaks. I am sure there is plenty of space for improvements. Please share your experience and add a comment (or two). Thank you.