Apache Kafka Complete Tutorial: From Beginner to Production
Apache Kafka is one of the most important technologies used in modern distributed systems, event-driven architectures, and microservices. At first, Kafka can feel complicated because it introduces many…
Apache Kafka is one of the most important technologies used in modern distributed systems, event-driven architectures, and microservices.
At first, Kafka can feel complicated because it introduces many new concepts:
- Broker
- Cluster
- Topic
- Partition
- Offset
- Producer
- Consumer
- Consumer Group
- Replication
- ISR
- Consumer Lag
- Rebalancing
- Idempotency
- Transactions
- Kafka Streams
- Kafka Connect
- KRaft
The good news is that these concepts are closely connected.
Once you understand the relationship between them, Kafka becomes much easier to reason about.
This tutorial takes a practical path from the basics of Kafka to production-level concepts and Spring Boot implementation.
Throughout the tutorial, we will use an e-commerce example:
Order Service
|
| OrderCreated
↓
Kafka
|
+--------------+--------------+
| | |
↓ ↓ ↓
Inventory Payment Notification
Service Service Service
By the end of this tutorial, you should understand:
- What Kafka is and why it exists
- How topics and partitions work
- How producers and consumers work
- How consumer groups provide scalability
- How offsets and commits work
- How replication provides fault tolerance
- Why duplicate processing happens
- How to design idempotent consumers
- How retries and dead-letter topics work
- What Kafka transactions actually solve
- How KRaft works
- Kafka Streams and Kafka Connect
- Kafka security and monitoring
- Kafka performance and partitioning
- How Kafka fits into a Spring Boot microservices architecture
- How to design a production-ready Kafka flow
1. What Is Apache Kafka?
Apache Kafka is a distributed event-streaming platform.
Kafka allows applications to:
- Publish events
- Store events durably
- Read events
- Process events
- Replay events
- Scale processing horizontally
- Replicate data for fault tolerance
For example, when a customer places an order, the Order Service can publish:
OrderCreated
Kafka stores that event.
Multiple independent services can then consume the same event:
Kafka
|
OrderCreated
|
+-----------+-----------+
| | |
↓ ↓ ↓
Inventory Payment Notification
The Order Service does not need to synchronously call every downstream service.
This reduces coupling and allows each consumer to process the event independently.
2. Why Do We Need Kafka?
Consider a traditional synchronous architecture:
Order Service
|
+----→ Inventory Service
|
+----→ Payment Service
|
+----→ Notification Service
|
+----→ Analytics Service
This architecture can work, but as the system grows, several problems appear.
2.1 Tight Coupling
The Order Service needs to know about every downstream service.
2.2 Failure Propagation
If Inventory is unavailable, the Order Service may experience:
- Timeouts
- Retries
- Circuit breaker activation
- Increased latency
- Failed requests
2.3 Scaling Problems
The Order Service is responsible for waiting for multiple downstream operations.
2.4 Difficult Event Replay
A normal HTTP request is not a durable event log that other applications can independently replay later.
Kafka changes the architecture:
Order Service
|
| OrderCreated
↓
Kafka
|
+----→ Inventory
|
+----→ Payment
|
+----→ Notification
|
+----→ Analytics
The producer and consumers become much more loosely coupled.
3. Event-Driven Architecture
Kafka is commonly used to implement event-driven systems.
An event represents something that has already happened.
Examples:
OrderCreated
PaymentCompleted
InventoryReserved
OrderShipped
UserRegistered
PaymentFailed
For example:
OrderCreated
means:
An order has been created.
A command is different.
ReserveInventory
means:
Please perform this action.
Both events and commands can technically be transported through Kafka, but event-driven systems commonly use Kafka to distribute events between services.
4. The Kafka Mental Model
Before learning individual Kafka components, remember this hierarchy:
Kafka Cluster
|
+---- Broker
+---- Broker
+---- Broker
|
↓
Topic
|
+-----+-----+
↓ ↓ ↓
P0 P1 P2
| | |
Records Records Records
| | |
Offsets Offsets Offsets
On the consumer side:
Consumer
|
Consumer Group
|
Partition Assignment
|
Committed Offset
|
Consumer Lag
The most important relationship is:
Cluster
↓
Brokers
↓
Topics
↓
Partitions
↓
Records
↓
Offsets
If you understand this hierarchy, most Kafka concepts become easier.
5. Kafka Cluster and Brokers
A Kafka cluster is a group of Kafka servers.
Each Kafka server is called a broker.
For example:
Kafka Cluster
+-----------------------------+
| |
| Broker 1 |
| Broker 2 |
| Broker 3 |
| |
+-----------------------------+
Multiple brokers provide:
- Scalability
- Fault tolerance
- Distributed storage
- Parallel processing
- High availability
A broker stores partition data and handles requests from producers, consumers, and other Kafka components.
A broker does not necessarily contain an entire topic.
Topic partitions are distributed across brokers.
6. Kafka Topics
A topic is a logical stream of events.
Examples:
orders
payments
inventory-events
notifications
user-events
Suppose the Order Service publishes events to:
order-events
The topic might contain:
OrderCreated
OrderCreated
OrderCancelled
OrderShipped
A topic is a logical abstraction.
The actual records are stored inside partitions.
order-events
|
+---- Partition 0
+---- Partition 1
+---- Partition 2
7. Kafka Partitions
A partition is an ordered, append-only log.
Suppose order-events has three partitions:
Partition 0
Offset 0 → OrderCreated(1001)
Offset 1 → OrderCreated(1004)
Offset 2 → OrderShipped(1001)
Partition 1
Offset 0 → OrderCreated(1002)
Offset 1 → OrderCreated(1005)
Partition 2
Offset 0 → OrderCreated(1003)
Offset 1 → OrderCreated(1006)
Kafka guarantees ordering within a partition.
Kafka does not provide one global ordering across all partitions.
This distinction is extremely important.
7.1 Why Does Kafka Use Partitions?
Partitions provide:
- Parallelism
- Scalability
- Distributed storage
- Per-partition ordering
For example:
Topic: orders
P0 → Consumer 1
P1 → Consumer 2
P2 → Consumer 3
P3 → Consumer 4
Multiple consumers can process different partitions concurrently.
8. Partition vs Replication
Partitioning and replication solve different problems.
8.1 Partitioning
Partitioning is primarily about:
- Scalability
- Parallel processing
- Distribution
- Ordering boundaries
8.2 Replication
Replication is primarily about:
- Fault tolerance
- Availability
- Durability
For example:
Topic
|
+---- Partition 0
| |
| +---- Replica Broker 1
| +---- Replica Broker 2
| +---- Replica Broker 3
|
+---- Partition 1
|
+---- Replica Broker 2
+---- Replica Broker 3
+---- Replica Broker 1
9. Kafka Records and Offsets
A Kafka record is the unit of data stored in a partition.
A record can contain:
- Key
- Value
- Timestamp
- Headers
- Partition
- Offset
An offset identifies the position of a record within a partition.
For example:
Partition 0
Offset 0 → Order A
Offset 1 → Order B
Offset 2 → Order C
Offset 3 → Order D
9.1 Are Offsets Global?
No.
Offsets are maintained independently for each partition.
Partition 0:
0
1
2
3
Partition 1:
0
1
2
3
Therefore, a record’s position is identified by the partition and offset.
Partition = 1
Offset = 25
9.2 Do Offsets Reset When Records Are Deleted?
No.
Suppose a partition contains:
0 1 2 3 4 5 6
After retention removes older records:
2 3 4 5 6
Kafka does not renumber the remaining records.
Offsets continue increasing as new records are appended.
10. Kafka Producers
A producer is an application that writes records to Kafka.
For example:
Order Service
|
| OrderCreated
↓
Kafka
A producer typically determines:
- Which topic to use
- Which key to use
- What value to send
- Serialization format
- Reliability settings
For example:
kafkaTemplate.send(
"order-events",
orderId.toString(),
orderCreatedEvent
);
The important part here is the key.
11. Kafka Message Keys
A Kafka record can have a key.
For example:
key = orderId
value = OrderCreated
The key can influence which partition receives the record.
Suppose:
Order 1001
Order 1001
Order 1001
Using 1001 as the key can ensure those records are routed to the same partition.
That is useful when ordering matters for a particular entity.
For example:
OrderCreated
OrderPaid
OrderShipped
For order 1001, you may want these events to remain ordered.
11.1 Does a Key Provide Global Ordering?
No.
The guarantee is still:
Ordering is maintained within a partition.
A key helps ensure related records are routed consistently, but it does not create global ordering across the entire topic.
12. Kafka Consumers
A consumer reads records from Kafka.
For example:
Kafka
|
↓
Inventory Service
A consumer:
- Polls Kafka
- Receives records
- Processes them
- Tracks its position
- Commits progress
A consumer does not remove a record from Kafka after processing it.
Kafka retention determines how long the record remains available.
13. Consumer Groups
A consumer group is a set of consumers working together to process a topic.
Suppose a topic has four partitions:
P0
P1
P2
P3
And a consumer group has four consumers:
Consumer 1 → P0
Consumer 2 → P1
Consumer 3 → P2
Consumer 4 → P3
This allows processing to happen in parallel.
13.1 What Happens If There Are More Consumers Than Partitions?
Suppose there are:
4 partitions
6 consumers
Only four consumers can actively consume partitions at that moment.
Two consumers will remain without a partition assignment.
This is why partition count is important when planning consumer scalability.
13.2 Can Two Consumer Groups Consume the Same Topic?
Yes.
For example:
Topic: order-events
|
+-----+-----+
| |
↓ ↓
inventory-group analytics-group
Each consumer group maintains its own offsets.
This allows different applications to independently consume the same events.
14. Consumer Offsets and Commits
Kafka tracks the progress of a consumer group using committed offsets.
Suppose:
Partition 0
0
1
2
3
4
5
If the consumer has processed through offset 3, its committed position indicates its progress.
If the application crashes, it can resume from the committed position.
This leads to an important distinction:
Current processing position
vs
Committed position
They are not necessarily identical at every moment.
15. auto.offset.reset
auto.offset.reset controls what a consumer should do when no valid committed offset exists.
Common values are:
earliest
latest
none
15.1 earliest
Start from the earliest available record.
Useful when a new consumer group needs to process existing data.
15.2 latest
Start from new records arriving after the consumer starts.
This is common when the application only cares about future events.
15.3 none
Throw an error if no valid offset exists.
This can be useful when silently choosing a starting position would be dangerous.
Important:
latest does not mean Kafka skips messages every time a consumer restarts.
If a valid committed offset exists, Kafka uses that committed position.
16. Consumer Lag
Consumer lag indicates how far behind a consumer group is from the latest available records.
Conceptually:
Latest offset
|
| ← Lag
|
Consumer committed offset
High lag can indicate:
- Slow processing
- Insufficient consumers
- Too few partitions
- Downstream database problems
- Network issues
- CPU or memory pressure
Consumer lag is one of the most important Kafka production metrics.
17. Kafka Replication
Replication allows Kafka to maintain multiple copies of partition data.
Suppose a partition has replication factor 3:
Partition 0
Leader
|
+---- Replica
|
+---- Replica
If one broker fails, another replica can become the leader.
This improves availability and fault tolerance.
18. Leader and Follower Replicas
Each partition has one leader replica and one or more follower replicas.
Producers and consumers normally interact with the partition leader.
Followers replicate the leader’s data.
For example:
Broker 1
|
+---- Partition 0 Leader
Broker 2
|
+---- Partition 0 Follower
Broker 3
|
+---- Partition 0 Follower
If the leader fails, Kafka can elect an eligible replica as the new leader.
19. ISR: In-Sync Replicas
ISR stands for:
In-Sync Replicas
These are replicas that are sufficiently caught up with the partition leader according to Kafka’s replication rules.
For example:
Partition 0
Leader → Broker 1
ISR → Broker 1, Broker 2, Broker 3
If Broker 3 falls significantly behind, it may leave the ISR.
ISR is important for understanding Kafka durability and producer acknowledgements.
20. Producer Acknowledgements
The acks setting controls how much acknowledgement the producer requires from Kafka.
20.1 acks=0
The producer does not wait for an acknowledgement.
This provides lower latency but weaker delivery guarantees.
20.2 acks=1
The leader acknowledges the record.
This provides stronger guarantees than acks=0, but replication may not yet have completed to all replicas.
20.3 acks=all
The leader waits for the required in-sync replicas according to the topic configuration.
This provides the strongest durability among the standard acknowledgement options.
For production systems, acks=all is commonly used together with appropriate replication and min.insync.replicas settings.
21. min.insync.replicas
min.insync.replicas defines the minimum number of replicas that must be in sync for certain writes to succeed when using acks=all.
For example:
Replication Factor = 3
min.insync.replicas = 2
acks = all
If only one replica remains in sync, Kafka can reject the write instead of accepting a record with weaker durability.
22. Producer Retries and Idempotency
Network failures can create ambiguous situations.
For example:
Producer
|
| send
↓
Kafka
|
| record stored
|
X acknowledgement lost
The producer may not know whether Kafka stored the record.
It retries.
Without idempotent producer behavior, a duplicate record can potentially be created.
Kafka supports idempotent producers to prevent duplicate writes caused by producer retries under supported conditions.
23. Producer Idempotency vs Consumer Idempotency
These are different problems.
23.1 Producer Idempotency
Protects against duplicate writes caused by producer retries.
Producer
↓
Kafka
23.2 Consumer Idempotency
Protects the business operation when the same Kafka event is delivered more than once.
Kafka
↓
Consumer
↓
Database
A system may need both.
24. Why Can Kafka Deliver the Same Message More Than Once?
Consider this sequence:
Kafka
|
↓
Consumer
|
↓
Update Database
|
X
Crash
Suppose the database update succeeded but the consumer crashed before committing its Kafka offset.
Kafka still believes the record has not been successfully committed.
After restart:
Kafka
|
↓
Consumer
|
↓
Same event again
This is a classic at-least-once processing scenario.
25. Consumer Idempotency
A common solution is to assign every event a unique event ID.
For example:
{
"eventId": "evt-12345",
"orderId": "1001",
"type": "OrderCreated"
}
The consumer stores processed event IDs.
Conceptually:
processed_events
event_id
---------
evt-12345
evt-12346
evt-12347
Before processing:
Does eventId already exist?
|
+---+---+
| |
Yes No
| |
Skip Process
|
↓
Record event ID
The important part is atomicity.
26. Making Idempotency Atomic
A dangerous implementation is:
1. Check event ID
2. Update business data
3. Save event ID
Two concurrent attempts can potentially pass the first check.
A stronger approach is to use a database constraint on the event ID.
For example:
processed_events
event_id UNIQUE
Then the database itself protects against duplicate processing.
In production systems, the idempotency record and the business update should generally be handled in the same database transaction when they belong to the same consistency boundary.
27. Offset Commit Strategies
Consumers need to decide when progress should be committed.
27.1 Auto Commit
Kafka can periodically commit offsets automatically.
This is simple, but the application has less precise control over the relationship between processing and committing.
27.2 Manual Commit
The application controls when offsets are committed.
A common conceptual flow is:
Poll record
↓
Process record
↓
Business operation succeeds
↓
Commit offset
This gives the application better control over processing semantics.
28. Kafka Delivery Semantics
Kafka systems are commonly discussed using three delivery models.
28.1 At-Most-Once
A record is processed zero or one time.
Possible consequence:
Message lost
28.2 At-Least-Once
A record is processed one or more times.
Possible consequence:
Duplicate processing
This is common in real-world Kafka applications.
28.3 Exactly-Once
Kafka provides mechanisms for exactly-once processing semantics within supported Kafka processing workflows.
However, exactly-once should not be misunderstood as:
Every external database or API call magically becomes exactly once.
If your consumer calls an external system, you still need appropriate application-level design.
29. Retry and Dead Letter Topics
A consumer may fail while processing a record.
For example:
Kafka
↓
Consumer
↓
Processing fails
Instead of immediately giving up, the application may retry.
A typical architecture is:
Main Topic
|
↓
Consumer
|
+---- success → Done
|
+---- failure → Retry
|
+---- success
|
+---- repeated failure
|
↓
DLT
A Dead Letter Topic, or DLT, stores records that could not be processed successfully after the configured retry strategy.
The DLT should not simply become a place where failed records disappear.
It should support:
- Investigation
- Alerting
- Root-cause analysis
- Controlled replay
- Operational recovery
30. Consumer Rebalancing
Kafka assigns partitions to consumers within a consumer group.
If the group membership changes, Kafka may redistribute partitions.
This is called a rebalance.
For example:
Before
Consumer 1 → P0
Consumer 2 → P1
Consumer 3 → P2
If Consumer 2 crashes:
After
Consumer 1 → P0
Consumer 3 → P1 + P2
Rebalancing allows the consumer group to continue processing.
However, frequent rebalances can hurt performance.
31. max.poll.interval.ms
One important consumer configuration is:
max.poll.interval.ms
It defines how long a consumer can go between successful calls to poll() before Kafka considers it unresponsive for group management purposes.
If your processing takes too long, the consumer may leave the group and trigger a rebalance.
This is especially important when consumers perform slow database or external API operations.
32. KRaft: Kafka Without ZooKeeper
Modern Kafka deployments use KRaft mode instead of the older ZooKeeper-based architecture.
KRaft stands for:
Kafka Raft metadata mode
Kafka uses its own Raft-based metadata quorum.
This removes the need for a separate ZooKeeper cluster for Kafka metadata management.
A simplified view is:
Kafka Cluster
|
+---- Controller
+---- Controller
+---- Controller
|
+---- Broker
+---- Broker
+---- Broker
The controller quorum manages Kafka cluster metadata and coordination.
For modern Kafka deployments, you should learn KRaft rather than building new systems around ZooKeeper.
33. Kafka APIs
Kafka exposes several important APIs.
33.1 Producer API
Used to publish records.
33.2 Consumer API
Used to read records.
33.3 Admin API
Used to manage Kafka resources programmatically.
For example:
- Create topics
- Describe topics
- Manage configurations
- Inspect cluster information
34. Kafka Streams
Kafka Streams is a client library for building stream-processing applications.
A normal consumer might do:
Kafka
↓
Consumer
↓
Business Logic
↓
Database
Kafka Streams allows you to build processing pipelines such as:
Orders
↓
Filter
↓
Transform
↓
Group
↓
Aggregate
↓
Output Topic
Kafka Streams is useful for applications that need continuous processing of event streams.
35. KStream vs KTable
A KStream represents a stream of events.
For example:
OrderCreated
OrderCreated
OrderCancelled
OrderShipped
A KTable represents the latest state associated with keys.
Conceptually:
order-1001 → SHIPPED
order-1002 → CANCELLED
order-1003 → CREATED
This distinction is important when designing Kafka Streams applications.
36. Kafka Connect
Kafka Connect is a framework for moving data between Kafka and external systems.
For example:
Database
↓
Source Connector
↓
Kafka
↓
Sink Connector
↓
Elasticsearch
This avoids writing custom producer or consumer applications for many common integration scenarios.
37. Source and Sink Connectors
37.1 Source Connector
Moves data into Kafka.
Database
↓
Source Connector
↓
Kafka
37.2 Sink Connector
Moves data from Kafka to another system.
Kafka
↓
Sink Connector
↓
Database / Elasticsearch / Object Storage
38. Kafka Retention
Kafka does not normally delete records simply because a consumer has processed them.
Instead, Kafka manages records using topic cleanup and retention policies.
Retention can be based on:
- Time
- Size
- Log compaction
For example:
Topic
|
+---- Records from today
+---- Records from yesterday
+---- Older records
Kafka can remove older records according to the configured retention policy.
39. Log Compaction
Log compaction keeps the latest record for a given key.
For example:
user-101 → ACTIVE
user-101 → PREMIUM
user-101 → SUSPENDED
After compaction, Kafka can retain the latest state:
user-101 → SUSPENDED
Compaction is useful for topics that represent the latest state of entities.
40. Tombstone Records
A tombstone is a record with a key and a null value.
For example:
key = user-101
value = null
In a compacted topic, this can represent deletion of the key’s current value.
41. Tiered Storage
Kafka environments may use tiered storage to move older data to remote storage while keeping active data closer to the brokers.
This can help separate long-term retention requirements from local broker storage constraints.
The exact capabilities and configuration depend on the Kafka distribution and deployment environment.
42. Compression
Kafka supports compression to reduce network and storage usage.
Common compression codecs include:
- gzip
- snappy
- lz4
- zstd
Compression can reduce the amount of data transferred between producers, brokers, and consumers, at the cost of CPU usage.
43. Kafka Transactions
Kafka transactions allow producers to atomically write multiple records and coordinate with Kafka’s transactional processing mechanisms.
They are especially useful in Kafka-to-Kafka processing pipelines where you need stronger guarantees around producing output records and committing consumed offsets.
For example:
Consume Event
↓
Process
↓
Produce Result
↓
Commit Kafka Progress
Transactions can make this workflow atomic within Kafka’s supported transactional model.
44. Kafka Transactions vs Database Transactions
This distinction is extremely important.
Suppose:
Kafka
↓
Consumer
↓
MySQL
A Kafka transaction does not automatically make the MySQL transaction atomic with Kafka.
You effectively have two systems:
Kafka transaction
+
Database transaction
Coordinating them requires an appropriate architecture.
This is why patterns such as the Transactional Outbox are important.
45. Transactional Outbox Pattern
Suppose the Order Service needs to:
- Save an order in the database
- Publish
OrderCreatedto Kafka
A naive implementation can fail:
Save order
↓
Database success
↓
Publish Kafka event
↓
Kafka failure
Now the order exists in the database, but the event was never published.
The Transactional Outbox pattern solves this by storing the business change and the event record in the same database transaction.
Order Service
|
+---- Order table
|
+---- Outbox table
Both are committed together.
A separate publisher then reads the outbox and publishes events to Kafka.
Database
|
↓
Outbox Publisher
|
↓
Kafka
This provides a reliable bridge between database state changes and Kafka events.
46. Eventual Consistency
In an event-driven microservice architecture, different services may not update at exactly the same moment.
For example:
Order Service
↓
Order = CREATED
Kafka
↓
Inventory Service
↓
Inventory = RESERVED
There may be a small period where the services have different views of the system.
This is eventual consistency.
Distributed systems often choose this model because strong cross-service transactions can be expensive and difficult to scale.
47. Saga Pattern
A Saga coordinates a distributed business transaction across multiple services.
For example:
Create Order
↓
Reserve Inventory
↓
Process Payment
↓
Confirm Order
If payment fails:
Payment Failed
↓
Release Inventory
↓
Cancel Order
Each service manages its own local transaction.
The overall business process is coordinated using events or commands.
48. Kafka Schema and Serialization
Kafka records contain serialized data.
Common formats include:
- JSON
- Avro
- Protobuf
- JSON Schema
For example:
{
"eventId": "evt-1001",
"orderId": "ORD-1001",
"customerId": "CUS-1001",
"amount": 1499.00
}
Serialization determines how application objects are converted into bytes and reconstructed by consumers.
49. Why Schema Management Matters
Without schema discipline, producers and consumers can easily become incompatible.
Suppose version 1 contains:
{
"orderId": "1001",
"amount": 100
}
Later, version 2 adds:
{
"orderId": "1001",
"amount": 100,
"currency": "INR"
}
Consumers need to handle such changes safely.
Schema management helps teams establish compatibility rules.
50. Event Versioning
Event contracts evolve over time.
A good event design considers:
- Backward compatibility
- Forward compatibility
- Optional fields
- Deprecated fields
- Versioning strategy
Avoid changing event contracts without considering every consumer.
Kafka decouples runtime communication, but it does not remove the need for contract management.
51. Kafka Headers
Kafka records can contain headers.
Headers can carry metadata such as:
correlationId
traceId
eventType
source
schemaVersion
This is particularly useful for distributed tracing and request correlation.
52. Kafka Security
Production Kafka deployments should address three major security areas:
Authentication
Authorization
Encryption
52.1 Authentication
Authentication determines:
Who is connecting?
Common mechanisms include SASL-based authentication and TLS client authentication.
52.2 Authorization
Authorization determines:
What is this client allowed to do?
For example:
inventory-service
↓
READ order-events
while another application may have permission to produce to that topic.
52.3 Encryption
TLS can protect data in transit between clients, brokers, and other Kafka components.
Security should be designed before exposing Kafka in a production environment.
53. Kafka Monitoring
Kafka requires monitoring at several levels.
53.1 Broker Metrics
Monitor:
- CPU
- Memory
- Disk
- Network
- Request latency
- Under-replicated partitions
53.2 Producer Metrics
Monitor:
- Record rate
- Error rate
- Retry rate
- Request latency
- Batch size
53.3 Consumer Metrics
Monitor:
- Consumer lag
- Processing rate
- Poll latency
- Rebalances
- Commit failures
54. Consumer Lag Monitoring
Consumer lag is one of the most important Kafka operational metrics.
For example:
Latest Offset = 100000
Committed Offset = 97000
Lag = 3000
A continuously increasing lag is usually more concerning than a temporarily high lag.
A production monitoring setup may use:
Kafka
↓
Metrics
↓
Prometheus
↓
Grafana
This allows teams to visualize Kafka health and create alerts.
55. Kafka Performance and Partitioning
More partitions are not automatically better.
Increasing partitions can improve parallelism, but it also increases:
- Metadata
- File handles
- Replication work
- Recovery work
- Operational complexity
Partition count should be chosen based on expected throughput, consumer parallelism, retention, and future growth.
56. Hot Partitions
A poorly chosen key can create a hot partition.
Suppose most events use the same key:
customerId = 1
Then many records may be routed to the same partition:
P0 → ███████████████████
P1 → ██
P2 → ██
P3 → █
The overall topic may have spare capacity while one partition becomes a bottleneck.
Partition-key design is therefore an important production decision.
57. Ordering vs Scalability
There is often a tradeoff between strict ordering and parallelism.
If all events must be globally ordered, using many partitions does not provide that guarantee.
Instead, design ordering around the entity that actually needs it.
For example:
orderId = 1001
OrderCreated
OrderPaid
OrderShipped
These events can be routed using the same key so they remain ordered within their partition.
58. Kafka and Database Consistency
A common microservice flow looks like this:
Kafka
↓
Consumer
↓
Database
The difficult part is coordinating:
Kafka offset
+
Database transaction
For example:
Process event
↓
Update database
↓
Crash
↓
Offset not committed
↓
Kafka redelivers event
This is why idempotency is so important.
A robust consumer should assume that duplicate delivery can happen.
59. Production Kafka Project: Order Processing System
Now let’s combine the concepts into one realistic Spring Boot microservices example.
The architecture is:
Client
|
↓
Order Service
|
| OrderCreated
↓
Kafka
|
+-----------+-----------+
| | |
↓ ↓ ↓
Inventory Payment Notification
Service Service Service
The example focuses on the relationship between Kafka, the consumer, the database, retries, idempotency, and failures.
59.1 Order Event
A simplified event could look like:
public record OrderCreatedEvent(
String eventId,
Long orderId,
Long customerId,
BigDecimal amount
) {
}
The eventId is important because it gives the consumer a unique identity for the event.
59.2 Spring Boot Producer
The Order Service can publish the event using Spring Kafka.
kafkaTemplate.send(
"order-events",
orderId.toString(),
event
);
Here:
Topic = order-events
Key = orderId
Value = OrderCreatedEvent
Using orderId as the key helps keep events belonging to the same order on the same partition.
59.3 Spring Boot Consumer
The Inventory Service can consume the event:
@KafkaListener(
topics = "order-events",
groupId = "inventory-group"
)
public void consume(OrderCreatedEvent event) {
// Process inventory
}
The consumer group:
inventory-group
allows multiple Inventory Service instances to share the work.
59.4 Production-Oriented Consumer Flow
A production consumer should not simply:
Receive
↓
Update DB
↓
Done
A more robust flow is:
Receive event
↓
Validate event
↓
Check idempotency
↓
Execute business transaction
↓
Record event as processed
↓
Commit Kafka progress
The exact implementation depends on the database and transaction boundaries, but the key idea is:
Make repeated delivery safe.
59.5 Handling Duplicate Events
Suppose:
OrderCreated(eventId=evt-123)
is delivered.
The consumer processes it:
Inventory updated
Event marked as processed
Then the application crashes before the Kafka offset is committed.
Kafka delivers the event again.
The consumer checks:
eventId = evt-123
The database says:
Already processed
The consumer can safely avoid repeating the business operation.
59.6 Idempotency Table
A simple table might contain:
processed_events
event_id
----------------
evt-123
evt-124
evt-125
The database should enforce uniqueness:
event_id UNIQUE
This prevents the same event from being recorded twice.
59.7 Retry and DLT in the Project
Suppose Inventory processing fails:
OrderCreated
↓
Inventory
↓
Database failure
The application can retry.
If the event continues to fail:
order-events
↓
Inventory Consumer
↓
Retry
↓
Retry
↓
Retry
↓
DLT
The DLT can then be monitored and investigated.
59.8 What Happens If Inventory Is Down?
Suppose:
Order Service
↓
Kafka
↓
Inventory Service ❌
The Order Service does not necessarily fail because Inventory is unavailable.
Kafka retains the event according to the topic’s retention policy.
When Inventory becomes healthy again:
Kafka
↓
Inventory
↓
Process pending events
The consumer can catch up.
This is one of the major advantages of asynchronous event-driven communication.
59.9 What Happens If the Consumer Crashes After Processing?
Consider:
Kafka
↓
Inventory Consumer
↓
Inventory Database
↓
SUCCESS
↓
Consumer crashes
↓
Offset not committed
Kafka may deliver the same event again.
Without idempotency:
Inventory update
Inventory update again
With idempotency:
First attempt
↓
Business operation
↓
eventId recorded
Second attempt
↓
eventId already exists
↓
Skip duplicate
This is one of the most important production Kafka failure scenarios to understand.
59.10 Kafka Replay
Because Kafka retains records independently of consumer processing, applications can replay events when appropriate.
For example:
Historical events
↓
New consumer group
↓
Process from earliest available offset
Replay can be useful for:
- Rebuilding derived state
- Reprocessing historical events
- Recovering from application bugs
- Bootstrapping new services
Replay should be designed carefully because business operations may not be naturally safe to repeat.
60. Kafka Internal Topics
Kafka uses internal topics for important cluster and consumer-group state.
For example, consumer group offsets are stored internally by Kafka.
You generally should not treat these internal topics as normal application topics.
Understanding them is useful when troubleshooting Kafka internals and consumer-group behavior.
61. MirrorMaker 2 and Disaster Recovery
For multi-cluster Kafka architectures, MirrorMaker 2 can replicate data between Kafka clusters.
A simplified architecture is:
Primary Kafka Cluster
|
| Replication
↓
Disaster Recovery Cluster
This can be used for:
- Disaster recovery
- Cross-region architectures
- Data migration
- Multi-cluster deployments
A real disaster recovery design must also consider:
- RPO
- RTO
- Network connectivity
- Topic configuration
- Consumer offsets
- Application failover
- Data consistency
62. Kafka Cluster Sizing
Kafka capacity planning should consider several dimensions.
62.1 Traffic
Estimate:
events/second
average event size
peak events/second
62.2 Storage
Estimate:
daily data
retention period
replication factor
62.3 Network
Consider:
producer traffic
consumer traffic
replication traffic
62.4 CPU
CPU usage depends on:
- Compression
- Encryption
- Request processing
- Replication
- Network workload
62.5 Disk
Kafka is heavily dependent on disk performance.
SSD storage is commonly preferred for production workloads that require high throughput and low latency.
62.6 Partitions
Partition count should support required throughput and consumer parallelism without creating unnecessary operational overhead.
63. Kafka Delivery Guarantees in a Real System
Consider the complete flow:
Producer
↓
Kafka
↓
Consumer
↓
Database
There are actually multiple delivery boundaries.
63.1 Producer to Kafka
Controlled using settings such as:
acks
retries
idempotence
63.2 Kafka to Consumer
Controlled through:
consumer group
offsets
commits
63.3 Consumer to Database
Controlled through:
database transactions
idempotency
application logic
This is why saying:
“Kafka guarantees exactly-once”
is incomplete.
You need to understand the entire system boundary.
64. Kafka Does Not Solve Everything
Kafka is powerful, but it is not the right solution for every problem.
Kafka is a good fit for:
- Event streaming
- High-throughput messaging
- Event-driven architectures
- Log-based processing
- Data pipelines
- Asynchronous microservice communication
- Event replay
Kafka may not be the best choice when you simply need:
- A small synchronous request
- A simple task queue
- Request-response communication
- A lightweight notification mechanism
Technology should follow the problem.
65. Kafka vs REST
REST is generally request-response oriented.
Client
↓
Service
↓
Response
Kafka is event-stream oriented.
Producer
↓
Kafka
↓
Consumers
REST is useful when the caller needs an immediate response.
Kafka is useful when events need to be processed asynchronously by one or more consumers.
They are complementary technologies, not necessarily competitors.
66. Kafka vs RabbitMQ
Kafka and RabbitMQ can both be used for asynchronous communication, but their architectural models differ.
66.1 Kafka
Kafka is particularly strong for:
- High-throughput event streaming
- Durable event logs
- Replay
- Partition-based scaling
- Stream processing
66.2 RabbitMQ
RabbitMQ is particularly strong for:
- Traditional messaging
- Routing
- Task queues
- Flexible delivery patterns
- Request/reply messaging
The correct choice depends on the problem being solved.
67. Common Kafka Mistakes
67.1 Using One Partition for Everything
This limits parallelism.
67.2 Increasing Partitions Without Understanding the Workload
More partitions create additional operational overhead.
67.3 Assuming Kafka Guarantees Global Ordering
Ordering is guaranteed within a partition.
67.4 Ignoring Consumer Lag
A consumer can appear healthy while continuously falling behind.
67.5 Assuming Kafka Prevents Duplicates
At-least-once processing can result in duplicate delivery.
Consumers should be designed accordingly.
67.6 Treating Kafka Transactions as Database Transactions
Kafka transactions do not automatically make external database operations atomic.
67.7 Using Poor Partition Keys
A bad key can create hot partitions and uneven load.
67.8 Sending Huge Messages Through Kafka
Large records can create memory, network, storage, and latency problems.
Large payloads should be evaluated carefully, and object storage may be more appropriate for large binary data.
68. Production Configuration Example
A Spring Boot producer might contain settings conceptually similar to:
spring:
kafka:
bootstrap-servers: localhost:9092
producer:
acks: all
retries: 10
properties:
enable.idempotence: true
A consumer might use:
spring:
kafka:
consumer:
group-id: inventory-group
auto-offset-reset: earliest
These settings are examples, not universal production defaults.
Production configuration should be chosen based on:
- Traffic
- Failure requirements
- Processing time
- Partition count
- Replication
- Infrastructure
- Business guarantees
69. Running Kafka Locally with Docker
For local development, Kafka can be run using Docker Compose.
A typical setup contains:
Kafka
|
+---- Broker
|
+---- Controller / KRaft
For modern Kafka development, KRaft-based configurations are generally preferable to introducing ZooKeeper into a new local setup.
Once Kafka is running, useful CLI operations include:
kafka-topics
kafka-console-producer
kafka-console-consumer
kafka-consumer-groups
These tools are extremely useful for understanding what Kafka is actually doing.
70. Inspecting Consumer Groups
One of the most useful operational commands is inspecting consumer groups.
Conceptually, the output contains:
GROUP
TOPIC
PARTITION
CURRENT-OFFSET
LOG-END-OFFSET
LAG
For example:
GROUP TOPIC PARTITION CURRENT END LAG
inventory-group order-events 0 9700 10000 300
This immediately tells you that the consumer is 300 records behind.
71. Complete Order Processing Flow
Let’s put everything together.
71.1 Step 1: User Places an Order
Client
↓
Order Service
71.2 Step 2: Order Service Saves the Order
Order Service
↓
Database
The order is stored successfully.
71.3 Step 3: OrderCreated Event Is Published
Order Service
↓
Kafka
Example:
eventId = evt-1001
orderId = ORD-1001
71.4 Step 4: Kafka Selects a Partition
The producer uses the order ID as the key.
key = ORD-1001
Kafka selects a partition based on its partitioning strategy.
71.5 Step 5: Kafka Assigns an Offset
The record receives an offset within that partition.
Partition 2
Offset 1842
71.6 Step 6: Kafka Replicates the Record
The partition’s replicas replicate the record according to the topic’s replication configuration.
71.7 Step 7: Inventory Consumes the Event
Kafka
↓
Inventory Consumer
71.8 Step 8: Inventory Processes the Event
The consumer:
Checks event ID
↓
Updates inventory
↓
Records processed event
71.9 Step 9: Consumer Commits Progress
After successful processing, the consumer commits its Kafka progress according to the configured processing model.
The complete flow is:
Client
↓
Order Service
↓
Database
↓
Kafka
↓
Inventory Consumer
↓
Idempotency Check
↓
Inventory Database
↓
Offset Commit
72. Complete Failure Scenario
Now consider the most important failure case.
Kafka
↓
Inventory Consumer
↓
Inventory Database
↓
SUCCESS
↓
Consumer crashes
↓
Offset not committed
Kafka may redeliver the event.
The consumer receives:
eventId = evt-1001
It checks the database:
evt-1001 already processed
Therefore:
Do not execute business operation again
This is why production Kafka systems must be designed around failure, not just the happy path.
73. Production Kafka Architecture
A realistic production architecture might look like:
API Gateway
|
↓
Order Service
|
↓
Database + Outbox
|
↓
Outbox Publisher
|
↓
Kafka Cluster
|
+--------------+--------------+
| | |
↓ ↓ ↓
Inventory Payment Notification
Service Service Service
| | |
↓ ↓ ↓
Database Database Provider
Operational components may include:
Kafka
↓
Prometheus
↓
Grafana
And distributed tracing can connect:
HTTP request
↓
Order Service
↓
Kafka
↓
Inventory Service
using correlation or trace identifiers.
74. What Should a Java Developer Know About Kafka?
You do not need to know every Kafka internals topic equally deeply.
A useful learning progression is:
74.1 Must-Know Kafka Concepts
- Kafka
- Broker
- Cluster
- Topic
- Partition
- Offset
- Producer
- Consumer
- Consumer Group
- Consumer Lag
- Kafka Key
74.2 Strong Intermediate Knowledge
- Replication
- Leader and follower
- ISR
acksmin.insync.replicas- Producer retries
- Idempotent producer
- Offset commits
- At-most-once
- At-least-once
- Retry
- DLT
- Rebalancing
74.3 Production-Level Knowledge
- Idempotent consumers
- Transactions
- Transactional Outbox
- Eventual consistency
- Saga
- Schema evolution
- Security
- Monitoring
- Partition strategy
- Consumer lag
- Disaster recovery
74.4 Advanced Knowledge
- Kafka Streams
- Kafka Connect
- KRaft
- Tiered storage
- Cluster sizing
- Multi-cluster architecture
- Performance tuning
75. Kafka Interview Questions You Should Be Able to Answer
After completing this tutorial, you should be able to answer questions such as:
75.1 Kafka Fundamentals
- What is Kafka?
- What is a broker?
- What is a Kafka cluster?
- What is a topic?
- What is a partition?
- What is an offset?
- Are Kafka offsets global?
- Why does Kafka use partitions?
75.2 Consumers
- What is a consumer group?
- How are partitions assigned to consumers?
- What happens when a consumer crashes?
- What happens when there are more consumers than partitions?
- What is consumer lag?
- What does
earliestmean? - What does
latestmean? - What is consumer rebalancing?
75.3 Producers
- What is a Kafka producer?
- What is a Kafka message key?
- Why would you use an order ID as the key?
- What does
acks=allmean? - What is producer idempotency?
- Why does Kafka retry producer requests?
75.4 Reliability
- What is at-most-once delivery?
- What is at-least-once delivery?
- Why can duplicate processing happen?
- How do you make a Kafka consumer idempotent?
- What is a Dead Letter Topic?
- When should you retry a Kafka message?
75.5 Architecture
- What is replication?
- What is ISR?
- What is KRaft?
- Does modern Kafka require ZooKeeper?
- What is Kafka Streams?
- What is Kafka Connect?
- What is the Transactional Outbox pattern?
- How would you handle Kafka and database consistency?
76. Kafka Cheat Sheet
Kafka
→ Distributed event-streaming platform
Broker
→ Kafka server
Cluster
→ Group of Kafka brokers
Topic
→ Logical stream of records
Partition
→ Ordered append-only log
Offset
→ Position within a partition
Producer
→ Writes records
Consumer
→ Reads records
Consumer Group
→ Consumers sharing partition-processing work
Key
→ Helps determine partition placement
Replication
→ Copies partition data for fault tolerance
ISR
→ In-sync replicas
Consumer Lag
→ Difference between latest data and consumer progress
Rebalancing
→ Redistribution of partitions among consumers
Idempotency
→ Making repeated processing safe
DLT
→ Destination for records that cannot be processed successfully
KRaft
→ Kafka's metadata quorum architecture
Kafka Streams
→ Stream-processing library
Kafka Connect
→ Framework for integrating Kafka with external systems
Outbox Pattern
→ Reliable bridge between database changes and Kafka events
77. The Kafka Mental Model to Remember
When thinking about Kafka, divide the system into five areas.
77.1 Data Side
Topic
↓
Partition
↓
Record
↓
Offset
77.2 Producer Side
Producer
↓
Key
↓
Partition
↓
Acknowledgement
77.3 Consumer Side
Consumer
↓
Consumer Group
↓
Partition Assignment
↓
Processing
↓
Offset Commit
77.4 Reliability Side
Replication
ISR
acks
Retries
Idempotency
Transactions
77.5 Production Side
Monitoring
Security
Schema Management
Retry / DLT
Disaster Recovery
Capacity Planning
78. The Most Important Kafka Failure Scenario
If you remember only one production scenario, remember this:
Kafka
↓
Consumer
↓
Database Update
↓
SUCCESS
↓
Consumer crashes
↓
Offset not committed
↓
Kafka redelivers
↓
Duplicate event
↓
Idempotency check
↓
Duplicate safely ignored
This single scenario connects several important Kafka concepts:
- Consumer offsets
- At-least-once delivery
- Failure handling
- Duplicate delivery
- Idempotency
- Database transactions
79. Final Kafka Architecture to Remember
A production-oriented event-driven system can look like:
Client
|
↓
API Gateway
|
↓
Order Service
/ \
/ \
Database Outbox
|
↓
Publisher
|
↓
Kafka Cluster
|
+-------------------+-------------------+
| | |
↓ ↓ ↓
Inventory Payment Notification
Consumer Consumer Consumer
| | |
↓ ↓ ↓
Database Database Provider
Around this architecture:
Security
Monitoring
Tracing
Schema Management
Retry
DLT
Idempotency
Disaster Recovery
all become part of the production design.
80. One Sentence That Explains Kafka
If you need a simple mental model for an interview, remember:
Kafka is a distributed, durable event log that allows producers to publish records and multiple independent consumer groups to process those records at scale.
Everything else builds on this foundation:
Topic
↓
Partitions
↓
Records
↓
Offsets
↓
Consumer Groups
↓
Replication
↓
Reliability
↓
Production Architecture
81. What to Learn Next
Once you understand Kafka fundamentals, the next step is not to memorize more Kafka terminology.
Build something.
A good Spring Boot microservices project should include:
Order Service
Inventory Service
Payment Service
Notification Service
|
↓
Kafka
Then implement these features one by one:
81.1 Producer
Build a Spring Boot producer and publish an OrderCreated event.
81.2 Consumer
Build an Inventory Service that consumes the event.
81.3 Consumer Groups
Run multiple Inventory Service instances and observe how Kafka distributes partitions.
81.4 Multiple Partitions
Create multiple partitions and experiment with keys and ordering.
81.5 Manual Offset Handling
Experiment with successful processing, failures, and offset commits.
81.6 Retry
Introduce temporary failures and configure retry handling.
81.7 Dead Letter Topic
Send permanently failed messages to a DLT.
81.8 Idempotent Consumer
Implement an event ID table and make duplicate processing safe.
81.9 Transactional Outbox
Connect a database transaction with reliable event publication.
81.10 Kafka Transactions
Experiment with Kafka transactions and understand their boundaries.
81.11 Schema Evolution
Change event schemas while keeping consumers compatible.
81.12 Monitoring
Monitor consumer lag, broker health, throughput, and errors.
81.13 Docker
Run Kafka and your microservices locally using Docker Compose.
81.14 Kubernetes
Deploy Kafka-based microservices to Kubernetes and understand scaling behavior.
81.15 Production Deployment
Finally, learn how to deploy, secure, monitor, and operate the system in a production environment.
The best way to become confident with Kafka is to stop treating it as a collection of interview definitions and start reasoning about what happens when the system succeeds, slows down, crashes, retries, and recovers.
That is where Kafka becomes a production skill rather than just a technology you have studied.
