Fending off buzzwords can feel like a full-time job, so having a reliable approach to assessing technologies is a critical piece of an engineer’s toolkit. With the rise of Kafka, more and more companies are trying to figure out why there has been so much momentum behind it and if they should be evaluating it.
By walking through a set of 4 questions, we will gain a better understanding of Kafka’s position in an organization’s tech stack.
This may seem like an obvious question to ask yourself but it’s important to know the real intent of a technology before making any assumptions about it.
Kafka behaves like a publisher/subscriber (pub/sub) system. Messages are sent to topics by producers and available for consumption by downstream consumers. The diagram below illustrates the architecture with a caveat around consumer behavior. Traditional messaging systems push messages to all subscribers immediately while Kafka flips this and forces consumers to poll for new data on their own terms. As we will see in later sections, this change in architecture leads to a handful of benefits.
Where Apache Kafka starts to differentiate itself is in some of the guarantees and properties that it provides –
The origination of a technology shouldn’t be the main driver behind adoption but it does help in gaining a sense of trust in its quality and longevity.
The origin story for Kafka comes courtesy of LinkedIn. Once again, being backed by a large company isn’t reason enough to invest in a technology but it does help to know that they have the resources to build out and support it. It’s also a good sign to see how heavily they have adopted it (LinkedIn hits 1 trillion messages per day). Also, Kafka’s open source presence continues to grow and companies like Confluent and others are forming with Kafka at the core of its business model.
It’s always interesting to dig into what problems the creator was trying to solve when the tool was built. It provides a quick start to thinking about the problems that it can solve within your organization but should by no means act as the guard rails to its potential applications.
Like most companies, LinkedIn had a need to break down data silos to enable better access and analytics across the board. Existing products in the market (which will be discussed in the next section) addressed some requirements better than others but none of them explicitly provided the guarantees that they needed. Furthermore, Kafka was developed towards the forefront of the “microservice” movement and played really nicely into enabling an event-driven architecture within the organization by decoupling services.
Rightfully so, this is usually the question that gets the most discussion. What makes this better than xyz?
To answer this, we have to look at Kafka from a few different angles to see who the competitors are.
When looking at the tools that are closest to Kafka they usually get bucketed as either a “queue” or a “pub/sub messaging” tool. Let’s look at both of these and then how Kafka fits in.
Most message queue systems act as storage for “in-flight” messages and once they’re consumed, they’re cleared out. Many of the benefits illustrated above regarding Kafka as a messaging system are due to that fact that it is first and foremost a great storage system.
Storage is at the core of Kafka with data being written to disk and replicated. Publishers of data can wait for different levels of guarantees (acknowledgments) that their data has been persisted AND replicated. Consumers completely drive their read position, so Kafka in a way is just managing a distributed file system of logs.
At the 2018 Kafka Summit, Martin Kleppman went as far as exploring the idea that Kafka could be considered a database due to many of the properties it exhibits.
Producing, storing, and consuming Kafka messages is great, but maybe not great enough to warrant a complete pivot in architecture to gain some of the benefits of Kafka.
Distributed file systems allow for static files to be used in batch processing (i.e. processing of historical data in batches). Traditional messaging systems allow for the processing of future messages that arrive after you are subscribed. Kafka combines both of these to allow for streams applications to process historical data as well as new data as it arrives (i.e. real time).
On top of the core Kafka components that enable the messaging and storage functionality, a huge selling point of Kafka is the streaming client library that it offers. This library solves a lot of the hard problems like handling out of order data, reprocessing, stateful computations, joining across data streams, etc. At its core, the Kafka Streams Processing API builds on top of the core Kafka primitives from utilizing consumers/producers, topics for stateful storage, and consumer groups for scalability. The amount of dogfooding that can be found around the Kafka ecosystem is a testament to their architecture.
The truth is if you’re serious about adopting a technology like Kafka you are not going to have a complete answer with just these 4 questions. However, you are in a much better position to continue evaluating or set it aside for another time.
To move forward with an evaluation, you might consider going deeper into –