Aug 14, 2018

3 Kotlin Features to Improve Your Kafka Connect Development

Kafka Connect is the hub that connects your Kafka cluster to any other system. The framework aims to make it easy to pull data into Kafka as well as copy data out of Kafka. As mentioned in a previous post on building a Kafka Connector with Gradle, there are a variety of open source connectors available for use but there are still use cases that will require the development of a custom connector.

If you find yourself starting from scratch, here are a few reasons to consider Kotlin as your language of choice.

1) Eliminate Boilerplate Code with Data Classes

Kafka connectors have a relatively simple task. They either fetch data from a source and push it to Kafka (source connector) or pull data from Kafka and sync it to a system (sink connector).

Both types of connectors require a fair amount of mapping data to and from domain models. Kotlin’s data classes provide a streamlined way of defining objects and leaving out the boilerplate code traditionally found in POJOs. Data classes also provide ways to copy and destructure objects should the need arise.

Here’s a comparison of a data class and a traditional POJO (that doesn’t include all of the properties/functionality that the data class has).

2) Cleanup Struct Conversion with Extension Methods

Many times the data source you want to interact with has a library you can use to jumpstart development. In this case, you won’t have control over the domain models and end up writing a variety of helper functions to convert the models to and from the Struct objects that Kafka expects so that Avro and the Schema Registry can be used.

Kotlin’s extension methods provide a nice way to tack on functionality to a class and keep your code clear of helper clutter.

Here’s an example of adding a toStruct method to a model pulled in from a 3rd party library.

3) Java Interoperability

Java is the dominant language in the Kafka space so being able to pull in Java libraries or experiment with Kotlin in a Java project is one of the most powerful features that Kotlin offers. If you are trying to introduce Kotlin to your team, there is no problem bringing it in one class at a time so that you can show off some of the other features that Kotlin users love.


Disclaimer: Once you start writing Kotlin you may start to notice more and more .kt files appearing in your project.

About the Author

Matt Schroeder profile.

Matt Schroeder

Director, Real-Time Data

A wide range of professional experience and a Master’s Degree in Software Engineering have become the foundation that enables Matt to lead teams to the best solution for every problem.

Leave a Reply

Your email address will not be published.

Related Blog Posts
Building Better Data Visualization Experiences: Part 1 of 2
Through direct experience with data scientists, business analysts, lab technicians, as well as other UX professionals, I have found that we need a better understanding of the people who will be using our data visualization products in order to build them. Creating a product utilizing data with the goal of providing insight is fundamentally different from a typical user-centric web experience, although traditional UX process methods can help.
Kafka Schema Evolution With Java Spring Boot and Protobuf
In this blog I will be demonstrating Kafka schema evolution with Java, Spring Boot and Protobuf.  This app is for tutorial purposes, so there will be instances where a refactor could happen. I tried to […]
Redis Bitmaps: Storing state in small places
Redis is a popular open source in-memory data store that supports all kinds of abstract data structures. In this post and in an accompanying example Java project, I am going to explore two great use […]
Let’s build a WordPress & Kernel updated AMI with Packer
First, let’s start with What is an AMI? An Amazon Machine Image (AMI) is a master image for the creation of virtual servers in an AWS environment. The machine images are like templates that are configured with […]