Functional Testing With Embedded Kafka

Spring publishes a spring-kafka-test library that is promoted as a way to do some unit testing with Kafka. The project I’m currently working on reads messages from a JMS queue, does some transforming of that data, then produces that message onto a Kafka topic for downstream consumers to use as they see fit. Since we do functional tests, in addition to unit tests, I needed to verify that certain transformations make it onto the topic in the expected format. The Spring Kafka Test library provides an embedded Kafka broker that works great for this. I’ll show you how I implemented it using Gradle and a Spring Boot application. A sample project is here.

To start the embedded Kafka instance and the Spring Boot application, I used the SpawnProcessTask as the task type. To actually start the Kafka broker, just call the before() method on the instance. This starts up an embedded Zookeeper and Kafka at the same time. The constructor to the KafkaEmbedded class has a few parameters, but there really isn’t much that can be configured beyond those few parameters. This task also starts up the Spring Boot application as an executable jar. You could run it through a bootRun Gradle task, but using the jar is closer to how it will run out in the wild.

Once the services are up and running, the functionalTest task finally takes over and runs the functional tests.

My functional tests are written using Spock, but any framework could probably be used. I created a base functional test class to house some common functionality. The big two methods in this case send the request to the REST endpoint and attempt to consume the produced message from Kafka. The consume() method takes a topic to consume from, the key value to look for, and a couple of parameters to control timing and retries. If you don’t use keys in your system, you could probably look for some other value in your consumed message to indicate the message you are looking for.

My simple test just sends a message to the web service and tries to consume it. Since a real service is probably transforming the message in some way, it would also make sense to verify those transforms are being performed correctly.

When the tests have completed, the stopServer task is executed which shuts down the Spring Boot service and the embedded Kafka instance with the after() method call.

With this basic framework, functional tests with Kafka are super easy and that makes developers want to write more tests!

(Check out the sample project)

About the Author

Brendon Anderson profile.

Brendon Anderson

Sr. Consultant

Brendon has over 15 years of software development experience at organizations large and small.  He craves learning new technologies and techniques and lives in and understands large enterprise application environments with complex software and hardware architectures.

One thought on “Functional Testing With Embedded Kafka

  1. Sandesh says:

    Hi Brendon,

    If I have to write Spock test script to mock Kafka Producer, how should I go ahead with

Leave a Reply to Sandesh Cancel reply

Your email address will not be published.

Related Blog Posts
Kafka Schema Evolution With Java Spring Boot and Protobuf
In this blog I will be demonstrating Kafka schema evolution with Java, Spring Boot and Protobuf.  This app is for tutorial purposes, so there will be instances where a refactor could happen. I tried to […]
Redis Bitmaps: Storing state in small places
Redis is a popular open source in-memory data store that supports all kinds of abstract data structures. In this post and in an accompanying example Java project, I am going to explore two great use […]
Let’s build a WordPress & Kernel updated AMI with Packer
First, let’s start with What is an AMI? An Amazon Machine Image (AMI) is a master image for the creation of virtual servers in an AWS environment. The machine images are like templates that are configured with […]
Single Node Kubernetes on Raspberry Pi 4
In this post, I will go through the steps I used to create a single node Kubernetes combination master and worker node on a Raspberry Pi 4. The heavy lifting in this post is done […]