Configuring Kafka Topics with Spring Kafka

Here’s a common scenario, your engineering team decides to use Kafka and they start writing producers, consumers, connectors, streams, you name it. Before long, 6 engineers have manually created 20+ topics in the lower environment and its time to promote that great work into the next environment. Some teams may be fine continuing down the path of manual management and assigning a “topic master” to handle all topic change requests. Other teams may not be fine with this and look towards automation.

I recently came across a scenario similar to this and during my research was surprised at the lack of solutions for managing a Kafka cluster’s topics. With Spring Kafka already in the mix, I started perusing their documentation and stumbled on a small section of the docs that talk about configuring topics via a NewTopic class. The rest of this post details my findings as well as a solution to managing topic configurations.

NewTopic Beans

To test out creating new topics via code I created a simple Spring Boot application and defined three NewTopic beans. I gave each topic a name, a number of partitions, and replication factor. You can also provide the NewTopic with a map of config values to set up any of the topic level configurations. To keep things simple I left this out.

After running the app, I jumped onto my local broker and listed out the topics and there they were. Easy!

The KafkaAdmin

Behind the scenes, there is a KafkaAdmin tapped into the afterSingletonsInstantiated lifecycle hook that pulls all NewTopic beans out of the Application Context and runs through some logic to figure out which topics need to be created and which need to be updated.

If the broker is running Kafka 1.0.0 or higher, the KafkaAdmin can increase a topic’s partitions. It will not decrease the number of partitions.

This was a great start to getting away from manual topic management but I still wanted to get the configuration out of the code and into properties to make it more flexible across environments as well as simpler to maintain.

Dynamically Creating NewTopic Beans

The first step of pushing the topic configuration out of the code was to come up with a yaml format that maps to a POJO. The below snippet shows what that format looks like as well as the @ConfigurationProperties model they map to.

Now that these properties are available for injection, I created a @Configuration class and wired them in. In the @PostConstruct, the topic configurations are iterated over and injected into the Application Context as a NewTopic bean so that they will get picked up by the normal KafkaAdmin logic mentioned above.

This project makes a Kafka Cluster more maintainable as well as secure by reducing the needs for direct broker access. It centralizes topic configurations and provides an audit trail of operations on a topic. Future environments can be quickly configured by adding a new application-{env}.yml file and running the app with that {env} as one of the active profiles.


Source Code: https://github.com/schroedermatt/spring-kafka-config

About the Author

Matt Schroeder profile.

Matt Schroeder

Director, Modern API

A wide range of professional experience and a Master’s Degree in Software Engineering have become the foundation that enables Matt to lead teams to the best solution for every problem.

One thought on “Configuring Kafka Topics with Spring Kafka

  1. TRWS says:

    How to achieve this with kafkatemplate?

  2. Victor says:

    This is really really useful and cool. Thanks

  3. Somebody says:

    Link to ” a small section of the docs” is broken

    1. Thanks. The link is updated.

    1. Thanks! The link is updated.

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Blog Posts
Interpreting Spatial Data in the Age of COVID-19
As 2020 has come to an end, many are eager to leave the mess of COVID-19 behind with the new year and gain a fresh start. Unfortunately, new cases are still soaring across the United […]
Building a Better Mousetrap
Recently, my daughter (age 6) was into building “mousetraps” out of shoe boxes. These were more or less comfortable cardboard mouse houses complete with beds, rooms, everything a mouse could want or need and not […]
ARM Wrestling Its Way Into Mainstream Software Development
Nearly all smart phones have been running ARM-based processors for years. They provide superior power for the amount of power consumed, and thus extend battery life. With Apple’s recent release of the Apple Silicon M1 […]
Video capture using WebRTC
Capturing the feed of a users camera can be a daunting undertaking involving one or multiple third party packages. Often we opt for prompting the user to upload the video or image we need, causing […]