Kafka: a Distributed Messaging System for Log Processing

The problem: everyone wanted every event, and each got their own pipe

LinkedIn in 2010 generated a torrent of activity data (page views, searches, profile edits, ad impressions) and a torrent of operational metrics, and a growing list of systems wanted all of it: the Hadoop warehouse for analytics, the search index for relevance, the news feed, security monitoring, dashboards. Each consumer was wired to each producer with its own pipeline, and the pipelines fell into two families that were each wrong in their own way. Log aggregators like Scribe and Flume shipped files to Hadoop in batches and were useless for anything real time. Message queues like ActiveMQ delivered individual messages with per-message acknowledgements and brokers that tracked the state of every message, and fell over at the throughput a website produces.

The idea: a durable, partitioned log that consumers read at their own pace

Kafka’s authors kept the one thing databases had learned (the previous two papers on this list) and discarded what message queues assumed. A topic is a set of partitions; each partition is an append-only log on disk, and a message’s position in it is its offset. The broker remembers nothing about who has read what. Consumers pull messages and keep their own offset, so a slow consumer costs the broker nothing, a crashed consumer resumes from where it was, and a brand-new system can start from offset zero and replay weeks of history. Messages are kept for a configured time regardless of consumption. The log, not the message, is the unit of design.

How it works

Everything in the design serves throughput. A partition is a set of segment files that are only ever appended, so writes are sequential. Reads rely on the operating system’s page cache instead of an in-process cache, which survives broker restarts, and use the sendfile system call to move bytes from disk to the network socket without passing through the application at all. Producers batch messages and choose a partition, usually by hashing a key, which gives the one ordering guarantee Kafka makes: messages with the same key, in the same partition, are delivered in the order they were written.

Consumers organise into groups. Every partition of a topic is read by exactly one member of each group, so a group scales out to the number of partitions and no further, and two groups reading the same topic are entirely independent. The paper’s benchmarks against ActiveMQ and RabbitMQ showed roughly an order of magnitude more throughput for both producing and consuming, with messages batched and nothing acknowledged individually. At-least-once delivery was the guarantee; exactly-once, through idempotent producers and transactions, came six years later.

What it changed

Kafka was open-sourced the year the paper appeared and became the default spine of data infrastructure: the thing every service writes to and every warehouse, index, cache and stream processor reads from. Jay Kreps’s essay “The Log”, written two years later, generalised the paper into an argument about system design that is still assigned reading. Change-data-capture tools turned database logs into Kafka topics; Kafka Streams and Flink turned the topics back into tables; event sourcing made the log the primary record. Rivals such as Pulsar, Redpanda and Kinesis compete on operations and cost, not on the model.

Where you meet it today

• Kafka itself, in most companies above a certain size, and its hosted forms.
• Consumer lag as the health metric of a data platform: the offset diagram above, as a dashboard.
• Replay as a feature. Rebuilding a search index or a cache from the log, instead of from a backup, is now expected.
• Ordering per key. Any system that promises “events for one user arrive in order” is making Kafka’s promise with Kafka’s mechanism, a partition chosen by key.