Functional Testing With Embedded Kafka

Spring publishes a spring-kafka-test library that is promoted as a way to do some unit testing with Kafka. The project I’m currently working on reads messages from a JMS queue, does some transforming of that data, then produces that message onto a Kafka topic for downstream consumers to use as they see fit. Since we do functional tests, in addition to unit tests, I needed to verify that certain transformations make it onto the topic in the expected format. The Spring Kafka Test library provides an embedded Kafka broker that works great for this. I’ll show you how I implemented it using Gradle and a Spring Boot application. A sample project is here.

To start the embedded Kafka instance and the Spring Boot application, I used the SpawnProcessTask as the task type. To actually start the Kafka broker, just call the before() method on the instance. This starts up an embedded Zookeeper and Kafka at the same time. The constructor to the KafkaEmbedded class has a few parameters, but there really isn’t much that can be configured beyond those few parameters. This task also starts up the Spring Boot application as an executable jar. You could run it through a bootRun Gradle task, but using the jar is closer to how it will run out in the wild.

Once the services are up and running, the functionalTest task finally takes over and runs the functional tests.

My functional tests are written using Spock, but any framework could probably be used. I created a base functional test class to house some common functionality. The big two methods in this case send the request to the REST endpoint and attempt to consume the produced message from Kafka. The consume() method takes a topic to consume from, the key value to look for, and a couple of parameters to control timing and retries. If you don’t use keys in your system, you could probably look for some other value in your consumed message to indicate the message you are looking for.

My simple test just sends a message to the web service and tries to consume it. Since a real service is probably transforming the message in some way, it would also make sense to verify those transforms are being performed correctly.

When the tests have completed, the stopServer task is executed which shuts down the Spring Boot service and the embedded Kafka instance with the after() method call.

With this basic framework, functional tests with Kafka are super easy and that makes developers want to write more tests!

(Check out the sample project)

About the Author

Brendon Anderson profile.

Brendon Anderson

Sr. Consultant

Brendon has over 15 years of software development experience at organizations large and small.  He craves learning new technologies and techniques and lives in and understands large enterprise application environments with complex software and hardware architectures.

One thought on “Functional Testing With Embedded Kafka

  1. Sandesh says:

    Hi Brendon,

    If I have to write Spock test script to mock Kafka Producer, how should I go ahead with

Leave a Reply

Your email address will not be published.

Related Blog Posts
Natively Compiled Java on Google App Engine
Google App Engine is a platform-as-a-service product that is marketed as a way to get your applications into the cloud without necessarily knowing all of the infrastructure bits and pieces to do so. Google App […]
Building Better Data Visualization Experiences: Part 2 of 2
If you don't have a Ph.D. in data science, the raw data might be difficult to comprehend. This is where data visualization comes in.
Unleashing Feature Flags onto Kafka Consumers
Feature flags are a tool to strategically enable or disable functionality at runtime. They are often used to drive different user experiences but can also be useful in real-time data systems. In this post, we’ll […]
A security model for developers
Software security is more important than ever, but developing secure applications is more confusing than ever. TLS, mTLS, RBAC, SAML, OAUTH, OWASP, GDPR, SASL, RSA, JWT, cookie, attack vector, DDoS, firewall, VPN, security groups, exploit, […]