Streamlining Console Output Verification with Kotest

Console logging is often perceived as a bad practice, but in specific contexts, it’s quite suitable. Consider a scenario of a workshop where printing output directly to the console enhances transparency and simplifies the project setup. Using a comprehensive logger could lead to unnecessary clutter. However, there’s a downside to directly printing to the console—it makes it challenging to verify the result. Well, at least so I thought until I accidentally stumbled upon this awesome article by Thijs Kaper. Although Spring Boot provides out-of-the-box support for output verification, it’s also possible, and sometimes preferable, to implement it independently. In this blog post, I’ll demonstrate how to create a custom spec in Kotest that you can leverage in your tests. This spec automatically captures console output and provides the result for verification.

In Thijs’ post, he shows that you can reassign both the standard and error output streams. This handy trick lets you easily capture the output and save it for later verification.

Table of Contents

• Using Spring Boot
• Custom Implementation with Kotest
• The Constructor Magic
• Implementation Breakdown
• Output Verification in Tests
• Summary

Using Spring Boot

Spring Boot uses this principle behind the scenes and let’s you verify the console logs in unit tests. Here is how you can leverage the feature in Kotlin.

import io.kotest.matchers.string.shouldContain
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.extension.ExtendWith
import org.springframework.boot.test.system.CapturedOutput
import org.springframework.boot.test.system.OutputCaptureExtension

@ExtendWith(OutputCaptureExtension::class)
class MyTest {
  
  fun `should contain the expected logs`(output: CapturedOutput) {
    // TODO Run code that logs into Std.out
    
    // Now verify the captured output
    output.out shouldContain "Hello world!"
  }
}

Leaving Spring Boot out of the equation, let’s explore how you can implement a similar mechanism by yourself.

Custom Implementation with Kotest

The first piece of the puzzle is a container class for the captured output.

import java.io.ByteArrayOutputStream

class CapturedOutput(
  private val outBuffer: ByteArrayOutputStream, 
  private val errBuffer: ByteArrayOutputStream
) {
    val out: String = 
        get() = outBuffer.toString()
    val err: String =
        get() = errBuffer.toString()
}

Next, let’s create a custom test spec that can be reused across multiple test files. Since I prefer to keep my tests concise, I decided to base my custom spec on StringSpec. Feel free to choose any spec you like.

import io.kotest.core.spec.style.StringSpec
import java.io.ByteArrayOutputStream
import java.io.PrintStream

class CapturedOutput(
  private val outBuffer: ByteArrayOutputStream, 
  private val errBuffer: ByteArrayOutputStream
) {
    val out: String = 
        get() = outBuffer.toString()
    val err: String =
        get() = errBuffer.toString()
}

abstract class ConsoleSpec(body: ConsoleSpec.(CapturedOutput) -> Unit = {}) : StringSpec({
    val originalOut = System.out
    val originalErr = System.err
    val outBuffer = ByteArrayOutputStream()
    val errBuffer = ByteArrayOutputStream()
    val capturedOutput = CapturedOutput(outBuffer, errBuffer)

    beforeSpec {
        System.setOut(PrintStream(outBuffer))
        System.setErr(PrintStream(outBuffer))
    }

    afterSpec {
        System.setOut(originalOut)
        System.setOut(originalErr)
    }

    beforeEach {
        outBuffer.reset()
        errBuffer.reset()
    }

    body(this as ConsoleSpec, capturedOutput)
})

As you can see, the spec makes use of lifecycle hooks to manipulate the System.out and System.err streams.

The Constructor Magic

Before I delve into implementation details I thought it would be a good idea to pause by explaining the constructor.

abstract class ConsoleSpec(body: ConsoleSpec.(CapturedOutput) -> Unit = {}) : StringSpec({ .. })

Each Kotest spec has its body. This is where all the tests are managed and run. Now, sharing state is nothing the framework would particularly embrace. There are valid reasons for why there shouldn’t be any shared state among the tests.

If you decompile the StringSpec this is what you get.

public abstract class StringSpec public constructor(body: io.kotest.core.spec.style.StringSpec.() -> kotlin.Unit = COMPILED_CODE) : io.kotest.core.spec.DslDrivenSpec, io.kotest.core.spec.style.scopes.StringSpecRootScope {
}

As you can see the body takes no arguments. Meaning, each test suite has to declare state as local variables within the body. While this guarantees a perfect test isolation, it can sometimes lead to tedious duplicates.

Here is how our implementation would look like if we used the built-in StringSpec.

class TestOne : StringSpec({
    val originalOut = System.out
    // etc.
    val capturedOutput = ..
    beforeSpec { .. }
    afterSpec { .. }
    beforeEach { .. }

})

Imagine we wanted to add an additional class, TestTwo, as another test suite. You would need to perform all the bootstrapping entirely from scratch again. This would lead to a significant amount of boilerplate code duplication.

That’s why our custom spec handles everything internally and makes the outcome available as a body argument.

ConsoleSpec(body: ConsoleSpec.(CapturedOutput) -> Unit = {})

Our test classes can therefore conveniently access the captured output without making any additional effort.

class TestOne : ConsoleSpec({ capturedOutput ->
  // Use the capturedOutput for verification
})

Let’s move on to explaining implementation details.

Implementation Breakdown

Initially, the spec captures the original streams, creates custom buffers and instantiates capturedOutput as a shared state that can be accessed by tests.

    val originalOut = System.out
    val originalErr = System.err
    val outBuffer = ByteArrayOutputStream()
    val errBuffer = ByteArrayOutputStream()
    val capturedOutput = CapturedOutput(outBuffer, errBuffer)

Next, the spec redirects the standard output and error streams before the test suite kicks in. Once all the tests are done, it reverts back to the original setup.

    beforeSpec {
        System.setOut(PrintStream(outBuffer))
        System.setErr(PrintStream(outBuffer))
    }

    afterSpec {
        System.setOut(originalOut)
        System.setOut(originalErr)
    }

Clearing the buffer in between the tests is an important step we should not forget about.

    beforeEach {
        outBuffer.reset()
        errBuffer.reset()
    }

Finally, we are ready pass the captured output as a shared state accessed by your tests.

body(this as ConsoleSpec, capturedOutput)

Output Verification in Tests

Using the new spec is easy. Unlike the underlying StringSpec, it exposes the captured Std.out and Std.err a shared state that can be safely accessed by individual tests for verification.

class MyTest : ConsoleSpec({ capturedOutput ->

    "should capture output" {
        println("Hello world")
        capturedOutput.out shouldContain "Hello world"
    }

    "should capture error" {
        System.err.println("Too bad")
        capturedOutput.err shouldContain "Too bad"
    }
})

Summary

We’ve explored how to capture console output by reassigning standard output and error streams. In addition, we’ve mastered the process of creating a reusable test template that exposes console output as an additional argument in our tests. This method significantly simplifies testing by eliminating a lot of unnecessary boilerplate code, making it both efficient and simple to use.

The example is available on GitHub.