Apache Kafka
A distributed event streaming platform designed for high-throughput, fault-tolerant, and scalable real-time data streaming.
Detailed Explanation
Kafka is a distributed commit log that stores streams of events. It is designed for high throughput (millions of messages per second), durability (messages persisted to disk), and scalability (horizontal scaling via partitions). Kafka is used for: event streaming, log aggregation, real-time analytics, and data pipelines.
Kafka concepts: topics (categories of messages), partitions (parallel units within a topic), producers (write messages), consumers (read messages), consumer groups (parallel processing), and brokers (Kafka servers). Kafka retains messages for a configurable time period, allowing consumers to replay events.
Why It Matters
Kafka is the industry standard for high-throughput event streaming. It powers real-time data pipelines at companies like LinkedIn, Netflix, and Uber.
Real-World Example
A ride-sharing app uses Kafka: every GPS ping, ride request, and payment event is written to Kafka topics. Real-time analytics dashboards consume these events to show live driver locations and estimated arrival times.
When to Use
For high-throughput event streaming, real-time data pipelines, log aggregation, and when you need event replay capability. Not for simple task queues (use RabbitMQ).
Advantages
- Millions of messages per second
- Durable and fault-tolerant
- Horizontal scaling via partitions
- Event replay capability
- Ecosystem of connectors (Kafka Connect)
Disadvantages
- Complex to set up and operate
- Requires Zookeeper (or KRaft)
- Higher latency than RabbitMQ for small messages
- Overkill for simple messaging needs
- Steep learning curve
Related Terms
Frequently Asked Questions
What is the difference between Kafka and RabbitMQ?
Kafka is a distributed event streaming platform (high throughput, replay, persistent). RabbitMQ is a traditional message queue (flexible routing, lower latency for small messages). Kafka for streaming; RabbitMQ for task queues.
What is a Kafka topic?
A topic is a category of messages, similar to a database table. Topics are divided into partitions for parallelism. Each partition is an ordered, immutable sequence of messages.
What is a consumer group?
A consumer group is a set of consumers that jointly consume a topic. Each partition is consumed by exactly one consumer in the group, enabling parallel processing. Multiple groups can consume the same topic independently.
How long does Kafka retain messages?
Configurable per topic. Default is 7 days. Retention can be time-based (delete after X days) or size-based (keep last X GB). Kafka does not delete messages after consumption—they can be replayed.
Is Kafka overkill for my project?
If you need simple background job processing, use RabbitMQ or SQS. If you need high-throughput event streaming, real-time analytics, or event replay, Kafka is worth the complexity. Start simple; add Kafka when you need it.