Configuring Kafka Topics with Spring Kafka

Here’s a common scenario, your engineering team decides to use Kafka and they start writing producers, consumers, connectors, streams, you name it. Before long, 6 engineers have manually created 20+ topics in the lower environment and its time to promote that great work into the next environment. Some teams may be fine continuing down the path of manual management and assigning a “topic master” to handle all topic change requests. Other teams may not be fine with this and look towards automation.

I recently came across a scenario similar to this and during my research was surprised at the lack of solutions for managing a Kafka cluster’s topics. With Spring Kafka already in the mix, I started perusing their documentation and stumbled on a small section of the docs that talk about configuring topics via a NewTopic class. The rest of this post details my findings as well as a solution to managing topic configurations.

NewTopic Beans

To test out creating new topics via code I created a simple Spring Boot application and defined three NewTopic beans. I gave each topic a name, a number of partitions, and replication factor. You can also provide the NewTopic with a map of config values to set up any of the topic level configurations. To keep things simple I left this out.

After running the app, I jumped onto my local broker and listed out the topics and there they were. Easy!

The KafkaAdmin

Behind the scenes, there is a KafkaAdmin tapped into the afterSingletonsInstantiated lifecycle hook that pulls all NewTopic beans out of the Application Context and runs through some logic to figure out which topics need to be created and which need to be updated.

If the broker is running Kafka 1.0.0 or higher, the KafkaAdmin can increase a topic’s partitions. It will not decrease the number of partitions.

This was a great start to getting away from manual topic management but I still wanted to get the configuration out of the code and into properties to make it more flexible across environments as well as simpler to maintain.

Dynamically Creating NewTopic Beans

The first step of pushing the topic configuration out of the code was to come up with a yaml format that maps to a POJO. The below snippet shows what that format looks like as well as the @ConfigurationProperties model they map to.

Now that these properties are available for injection, I created a @Configuration class and wired them in. In the @PostConstruct, the topic configurations are iterated over and injected into the Application Context as a NewTopic bean so that they will get picked up by the normal KafkaAdmin logic mentioned above.

This project makes a Kafka Cluster more maintainable as well as secure by reducing the needs for direct broker access. It centralizes topic configurations and provides an audit trail of operations on a topic. Future environments can be quickly configured by adding a new application-{env}.yml file and running the app with that {env} as one of the active profiles.


Source Code: https://github.com/schroedermatt/spring-kafka-config

About the Author

Matt Schroeder profile.

Matt Schroeder

Director, Modern API

A wide range of professional experience and a Master’s Degree in Software Engineering have become the foundation that enables Matt to lead teams to the best solution for every problem.

One thought on “Configuring Kafka Topics with Spring Kafka

  1. TRWS says:

    How to achieve this with kafkatemplate?

  2. Victor says:

    This is really really useful and cool. Thanks

  3. Somebody says:

    Link to ” a small section of the docs” is broken

    1. Thanks. The link is updated.

    1. Thanks! The link is updated.

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Blog Posts
Google Professional Machine Learning Engineer Exam 2021
Exam Description A Professional Machine Learning Engineer designs, builds, and productionizes ML models to solve business challenges using Google Cloud technologies and knowledge of proven ML models and techniques. The ML Engineer is proficient in all aspects […]
Designing Kubernetes Controllers
There has been some excellent online discussion lately around Kubernetes controllers, highlighted by an excellent Speakerdeck presentation assembled by Tim Hockin. What I’d like to do in this post is explore some of the implications […]
React Server Components
The React Team recently announced new work they are doing on React Server Components, a new way of rendering React components. The goal is to create smaller bundle sizes, speed up render time, and prevent […]
Jolt custom java transform
Jolt is a JSON to JSON transformation library where the transform is defined in JSON. It’s really good at reorganizing the json data and massaging it into the output JSON you need. Sometimes, you just […]