Functional Testing With Embedded Kafka

Spring publishes a spring-kafka-test library that is promoted as a way to do some unit testing with Kafka. The project I’m currently working on reads messages from a JMS queue, does some transforming of that data, then produces that message onto a Kafka topic for downstream consumers to use as they see fit. Since we do functional tests, in addition to unit tests, I needed to verify that certain transformations make it onto the topic in the expected format. The Spring Kafka Test library provides an embedded Kafka broker that works great for this. I’ll show you how I implemented it using Gradle and a Spring Boot application. A sample project is here.

To start the embedded Kafka instance and the Spring Boot application, I used the SpawnProcessTask as the task type. To actually start the Kafka broker, just call the before() method on the instance. This starts up an embedded Zookeeper and Kafka at the same time. The constructor to the KafkaEmbedded class has a few parameters, but there really isn’t much that can be configured beyond those few parameters. This task also starts up the Spring Boot application as an executable jar. You could run it through a bootRun Gradle task, but using the jar is closer to how it will run out in the wild.

Once the services are up and running, the functionalTest task finally takes over and runs the functional tests.

My functional tests are written using Spock, but any framework could probably be used. I created a base functional test class to house some common functionality. The big two methods in this case send the request to the REST endpoint and attempt to consume the produced message from Kafka. The consume() method takes a topic to consume from, the key value to look for, and a couple of parameters to control timing and retries. If you don’t use keys in your system, you could probably look for some other value in your consumed message to indicate the message you are looking for.

My simple test just sends a message to the web service and tries to consume it. Since a real service is probably transforming the message in some way, it would also make sense to verify those transforms are being performed correctly.

When the tests have completed, the stopServer task is executed which shuts down the Spring Boot service and the embedded Kafka instance with the after() method call.

With this basic framework, functional tests with Kafka are super easy and that makes developers want to write more tests!

(Check out the sample project)

About the Author

Brendon Anderson profile.

Brendon Anderson

Sr. Consultant

Brendon has over 15 years of software development experience at organizations large and small.  He craves learning new technologies and techniques and lives in and understands large enterprise application environments with complex software and hardware architectures.

One thought on “Functional Testing With Embedded Kafka

  1. Sandesh says:

    Hi Brendon,

    If I have to write Spock test script to mock Kafka Producer, how should I go ahead with

Leave a Reply to Sandesh Cancel reply

Your email address will not be published. Required fields are marked *

Related Blog Posts
TICK Stack Monitoring for the Non-Technical
TICK – Telegraf, Influx, Chronograf, and Kapacitor – is a method of monitoring your systems and applications. In this article, I discuss in non-technical terms what the difference is between TICK and Prometheus Grafana A […]
Design Systems, Part 1 • Introduction
Business leaders need a practical guide to plan and execute Design System Initiatives. The aim of this series is to be that guide. This installment introduces terms and definitions as a primer on Design Systems.
ML for Translating Dysarthria Speech (Pre-Part 1)
What is Dysarthria? Per the Mayo Clinic, Dysarthria occurs when the muscles you use for speech are weak or you have difficulty controlling them. Dysarthria often causes slurred or slow speech that can be difficult […]
Develop Quality Code
As software continues to dominate every facet of our lives, developers are faced with an ever-increasing pressure to produce bug free code. The responsibility of clean quality software falls upon everyone that is involved in […]