Kafka Interview Preparation

Apache Kafka is a distributed event streaming platform developed by LinkedIn and later open-sourced under the Apache Software Foundation.

It is mainly used for:

• Real-time messaging
• Event streaming
• Log aggregation
• Data pipelines
• Event-driven Microservices
• Streaming analytics

Unlike a traditional message queue, Kafka stores messages on disk and allows consumers to read them multiple times.

Think of Kafka as a highly scalable distributed commit log.

Example

An E-commerce application

Order Service
      |
      | Order Created Event
      |
   Kafka Topic
      |
-------------------------
|          |            |
Inventory  Payment   Notification

One event can be consumed by multiple services.

Real-world Example

Amazon order placed

Instead of directly calling every service,

Order Service
      |
      |
Kafka
 |    |     |
Inventory
Payment
Shipping
Analytics
Notification

Every service receives the event independently.

Interview One-liner

u003e Apache Kafka is a distributed event streaming platform used for high-throughput, fault-tolerant, real-time data processing.

Kafka solves several problems in distributed systems.

Without Kafka

Order Service

 |--u003e Inventory

 |--u003e Payment

 |--u003e Email

 |--u003e Analytics

 |--u003e Shipping

Problems

• Tight coupling
• High latency
• Failure propagation
• Hard to scale

With Kafka

Order Service

      |

Kafka Topic

 |   |   |   |

Inventory

Payment

Shipping

Analytics

Advantages

• Loose coupling
• Asynchronous communication
• High throughput
• Fault tolerance
• Scalability
• Replay events
• Buffer traffic spikes

Interview Tip: Instead of multiple REST calls, one event can notify many services.

Example

RabbitMQ

Payment Queue

Message

↓

Consumer

↓

Deleted

Kafka

Topic

Message 1

Message 2

Message 3

Consumer A

Consumer B

Consumer C

All can read

Interview Answer

Use Kafka for

• Event streaming
• Analytics
• Logging
• Big data
• Microservices

Use RabbitMQ for

• Task queues
• Email queue
• Background jobs

Interview Answer

ActiveMQ is ideal for enterprise messaging and JMS-based applications, while Kafka is designed for distributed event streaming and high-volume real-time processing.

REST Example

Order

↓

Inventory

↓

Payment

↓

Notification

↓

Response

Every service must respond.

Kafka

Order

↓

Kafka

↓

Inventory

↓

Payment

↓

Notification

Producer finishes immediately after publishing the event.

Interview Answer

REST is best for request-response scenarios, while Kafka is ideal for asynchronous event-driven communication.

Use Kafka when

• Multiple services need the same event.
• High throughput is required.
• The producer should not wait for consumers.
• Temporary consumer downtime should not cause data loss.
• Event replay is needed.
• Loose coupling between services is desired.

Use REST when

• An immediate response is required.
• CRUD operations are performed.
• User-facing requests need instant confirmation.
• Strong request-response semantics are necessary.

Interview Scenario

Order Service needs to notify:

• Inventory
• Payment
• Shipping
• Analytics
• Notification

Instead of making five REST calls, publish a single OrderCreated event to Kafka. Every interested service consumes it independently.

Architecture Diagram

            Producer

                |

                |

         Kafka Cluster

      -------------------

      Broker 1

      Broker 2

      Broker 3

      -------------------

                |

             Topic

      Partition 0

      Partition 1

      Partition 2

                |

      Consumer Group

     Consumer A

     Consumer B

     Consumer C

Components

• Producer publishes messages.
• Broker stores messages.
• Topic categorizes messages.
• Partition enables scalability and parallelism.
• Consumer reads messages.
• Consumer Group distributes partitions among consumers.

Interview Tip: Remember the flow:

Producer → Topic → Partition → Broker → Consumer Group → Consumer

Topic

A Topic is a logical channel where related messages are stored.

Example:

order-topic

payment-topic

email-topic

Partition

A Topic is divided into multiple partitions.

Order Topic

Partition 0

Partition 1

Partition 2

Partitions provide

• Parallel processing
• Scalability
• Ordering within each partition

Offset

Each message in a partition has a unique sequential number called an Offset.

Partition 0

Offset 0

Offset 1

Offset 2

Offset 3

Consumers track the offset to know which messages have already been processed.

Interview Tip: Ordering is guaranteed only within a partition, not across partitions.

A Producer is an application that sends messages to Kafka topics.

Example

Order Service

↓

Producer

↓

Kafka Topic

Spring Boot Example

kafkaTemplate.send("order-topic", orderEvent);

Producer Responsibilities

• Serialize data.
• Choose partition.
• Compress messages (optional).
• Retry on failures.
• Wait for acknowledgments based on configuration.

A Consumer reads messages from Kafka topics.

Example

Kafka

↓

Consumer

↓

Inventory Service

Spring Boot Example

@KafkaListener(topics = "order-topic")
public void consume(OrderEvent event) {
    System.out.println(event);
}

Responsibilities

• Read messages.
• Deserialize payload.
• Process business logic.
• Commit offsets.

A Consumer Group is a set of consumers working together to consume messages from the same topic.

Rules:

• One partition is consumed by only one consumer within a group.
• Different consumer groups can independently consume the same topic.

Example:

Topic

Partition 0 → Consumer A

Partition 1 → Consumer B

Partition 2 → Consumer C

Benefit

Enables horizontal scaling and fault tolerance.

A Broker is a Kafka server responsible for storing partitions and serving producer and consumer requests.

A Kafka cluster typically contains multiple brokers.

Example:

Broker 1

Broker 2

Broker 3

Benefits:

• Scalability
• Fault tolerance
• High availability

ZooKeeper was an external coordination service used in earlier Kafka versions.

Responsibilities:

• Broker registration
• Leader election
• Cluster metadata
• Configuration management
• Health monitoring

Limitation

Maintaining ZooKeeper added operational complexity because it required a separate distributed system.

Interview Tip: ZooKeeper is not required in modern Kafka deployments using KRaft mode.

KRaft (Kafka Raft Metadata mode) replaces ZooKeeper by storing cluster metadata inside Kafka itself using the Raft consensus algorithm.

Benefits:

• No external ZooKeeper dependency
• Simpler deployment
• Faster metadata operations
• Better scalability
• Easier cluster management

Interview Tip: Kafka 4.x supports only KRaft mode, while Kafka 3.x supports both ZooKeeper and KRaft (depending on the version and configuration).

Kafka achieves very high throughput using multiple design optimizations:

• Sequential disk writes instead of random writes.
• Partitioning for parallel reads and writes.
• Batch processing to reduce network overhead.
• Zero-copy transfer using the operating system.
• Efficient binary protocol.
• Compression (GZIP, Snappy, LZ4, ZSTD).
• Asynchronous producers.
• Page cache utilization by the operating system.

Interview One-liner

u003e Kafka achieves high throughput through sequential disk I/O, batching, partitioning, zero-copy transfer, compression, and asynchronous processing.

Components

Producer
   │
   ▼
Topic
   │
Partitions
   │
Broker Cluster
   │
Replication
   │
Consumer Group

Flow

1. Producer sends a record.

2. Partitioner selects the target partition.

3. The leader replica stores the message.

4. Followers replicate it.

5. The leader acknowledges the producer (based on acks configuration).

6. Consumers fetch messages and commit offsets after processing.

Steps

1. Producer serializes the key and value.

2. Selects a partition:

• Explicit partition
• Key-based hashing
• Round-robin (if no key)

3. Batches records for efficiency.

4. Sends the batch to the leader broker.

5. Waits for acknowledgment based on:

• acks=0
• acks=1
• acks=all

Interview Tip: A producer always writes to the leader replica, never directly to followers.

Steps

1. Consumer joins a consumer group.

2. Kafka assigns partitions.

3. Consumer polls records using poll().

4. Processes the records.

5. Commits offsets (automatic or manual).

6. Polls again for new messages.

Spring Boot Example

@KafkaListener(topics = "order-topic")
public void consume(OrderEvent event) {
    // Business logic
}

1. Producer serializes the message.

2. Chooses the destination partition.

3. Sends the request to the partition leader.

4. Leader appends the message to the log.

5. Followers replicate the message.

6. Once the required replicas acknowledge (depending on acks), Kafka sends a success response to the producer.

Internal Flow

Producer
   │
Partition Selection
   │
Leader Broker
   │
Append to Log
   │
Followers Replicate
   │
ACK to Producer

Kafka stores data as an append-only log on disk.

Example:

Partition-0

Offset 0

Offset 1

Offset 2

Offset 3

New records are appended to the end of the log and are never updated in place.

Benefits

• Fast sequential writes
• Simple recovery
• High throughput
• Supports replay

Once written, Kafka messages cannot be modified.

Reasons:

• Preserves event history
• Simplifies replication
• Prevents concurrent update issues
• Improves performance
• Enables deterministic replay

If data changes, applications publish a new event instead of updating the old one.

Each partition log is divided into multiple segment files.

Example:

Partition

segment-000.log

segment-001.log

segment-002.log

Benefits:

• Easier deletion based on retention policies
• Faster log cleanup
• Efficient disk management

Each record in a partition receives an increasing offset.

Consumers store the last processed offset so they can resume after a restart.

Example:

Offset

0

1

2

3

4

If a consumer has committed offset 3, it resumes processing from offset 4.

Interview Tip: Offsets are maintained per consumer group, allowing different groups to consume the same topic independently.

Log compaction keeps only the latest value for each key while preserving the log structure.

Example:

User-101 -u003e Active

User-101 -u003e Inactive

User-101 -u003e Premium

After compaction:

User-101 -u003e Premium

Use Cases

• User profiles
• Product catalog
• Configuration data
• Cache rebuilding
• CDC (Change Data Capture)

Kafka removes old records according to configured retention policies.

Retention can be based on:

• Time (e.g., 7 days)
• Size (e.g., 100 GB)

Even if a consumer has already processed a message, Kafka retains it until the retention policy expires.

Interview Tip: Retention is independent of message acknowledgment.

ISR (In-Sync Replica) is the set of replicas that are fully caught up with the leader.

Example:

Leader

Follower 1 ✅

Follower 2 ✅

Follower 3 ❌ (lagging)

ISR:

Leader

Follower 1

Follower 2

Only replicas in the ISR are eligible for leader election.

Every partition has:

• One Leader
• Zero or more Followers

Leader

• Handles all reads and writes.
• Accepts producer requests.
• Serves consumer fetch requests.

Follower

• Replicates data from the leader.
• Does not serve writes.
• Can become the new leader if the current leader fails and it is part of the ISR.

Scenario

Suppose a partition has:

Leader : Broker 1

Followers :

Broker 2

Broker 3

If Broker 1 fails:

1. Kafka controller detects the failure.

2. It checks the ISR.

3. One in-sync follower (Broker 2 or Broker 3) is promoted to leader.

4. Producers and consumers automatically switch to the new leader.

Interview Tip: Kafka elects a new leader only from the ISR to avoid data loss. If unclean leader election is enabled, an out-of-sync replica may become leader, but this can result in data loss and is generally discouraged in production.

In Spring Boot, Kafka producers are created using Spring Kafka. The main class used to send messages is KafkaTemplate.

Step 1: Add Dependency

u003cdependencyu003e
    u003cgroupIdu003eorg.springframework.kafkau003c/groupIdu003e
    u003cartifactIdu003espring-kafkau003c/artifactIdu003e
u003c/dependencyu003e

Step 2: Configure Producer

spring.kafka.bootstrap-servers=localhost:9092

spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer

spring.kafka.producer.value-serializer=org.springframework.kafka.support.serializer.JsonSerializer

Step 3: Send Message

@Service
public class OrderProducer {

    @Autowired
    private KafkaTemplateu003cString, OrderEventu003e kafkaTemplate;

    public void publish(OrderEvent event) {
        kafkaTemplate.send("order-topic", event);
    }
}

Internal Flow

Spring Boot

↓

KafkaTemplate

↓

Kafka Producer API

↓

Broker Leader

↓

Kafka Topic

Interview Tip: In production, we usually send Java Objects instead of plain Strings. Spring automatically converts them into JSON using JsonSerializer.

KafkaTemplate is the primary class provided by Spring Kafka for sending messages to Kafka.

It acts as a wrapper around the native Kafka Producer API.

Common Methods

kafkaTemplate.send("order-topic", order);

kafkaTemplate.send("order-topic", key, order);

kafkaTemplate.sendDefault(order);

kafkaTemplate.flush();

Features

• Asynchronous sending
• Supports callbacks
• Automatic serialization
• Batch sending
• Transactions
• Future-based result handling

Example with Callback

kafkaTemplate.send("order-topic", order)
        .whenComplete((result, ex) -u003e {

            if (ex == null) {
                System.out.println("Message Sent");
            } else {
                System.out.println("Failed");
            }

        });

Interview One-liner

u003e KafkaTemplate is Spring Boot’s abstraction over the Kafka Producer API used for asynchronously publishing messages.

Kafka producers determine the target partition using the following order:

kafkaTemplate.send("order-topic", 2, "key", order);

Always goes to Partition 2.

If a key is provided,

kafkaTemplate.send("order-topic",
                   "USER101",
                   order);

Kafka computes

hash(key) % numberOfPartitions

The same key always maps to the same partition.

If no key is provided,

kafkaTemplate.send("order-topic", order);

Kafka distributes messages evenly across partitions.

Interview Tip: Partition selection order:

• Explicit partition
• Key hashing
• Round Robin (or sticky partitioning in newer Kafka clients)

When no key is provided, Kafka distributes messages across available partitions to balance the load.

Example:

Topic

Partition 0

Partition 1

Partition 2

Messages:

Message1 → P0

Message2 → P1

Message3 → P2

Message4 → P0

Message5 → P1

Advantages

• Uniform load distribution
• High throughput
• No hotspot partition

Limitation

Ordering is not guaranteed because related messages may go to different partitions.

Interview Tip: Modern Kafka clients often use a Sticky Partitioner (KIP-480) for better batching efficiency when no key is supplied, but the concept of balancing messages across partitions is still important for interviews.

When a key is provided, Kafka hashes the key to decide the partition.

Formula

hash(key) % totalPartitions

Example

User A

hash(User A)

↓

Partition 1

Every message with

User A

will always go to

Partition 1

Why?

Because ordering is guaranteed only inside one partition.

For example,

Order Created

Order Paid

Order Delivered

must stay in order.

Using

Order ID

as the key ensures all events go to the same partition.

Real-world Example

Bank Account

Account 1001

Deposit

Withdraw

Transfer

All account events should remain ordered.

Interview Tip: Use the business identifier (Order ID, Customer ID, Account ID, etc.) as the message key when event ordering is required.

Kafka allows developers to create custom partitioning logic.

Example:

public class RegionPartitioner implements Partitioner {

    @Override
    public int partition(String topic,
                         Object key,
                         byte[] keyBytes,
                         Object value,
                         byte[] valueBytes,
                         Cluster cluster) {

        String region = (String) key;

        if ("INDIA".equals(region))
            return 0;

        return 1;
    }
}

Configure

spring.kafka.producer.properties.partitioner.class=com.demo.RegionPartitioner

Use Cases

• Region-wise routing
• Premium customers
• Tenant-based routing
• Priority queues
• Country-specific processing

Interview Tip: Avoid custom partitioners unless the default strategies cannot satisfy business requirements.

acks defines how many brokers must acknowledge a message before the producer considers the send successful.

It directly impacts:

• Reliability
• Performance
• Data durability

Values

acks=0

acks=1

acks=all

Interview Tip: Higher reliability generally means higher latency.

Example

Leader + 2 Followers

Leader

Follower

Follower

acks=0

Producer sends and immediately returns.

No guarantee of delivery.

acks=1

Leader stores data.

Producer receives acknowledgment.

If the leader crashes before followers replicate, the message may be lost.

acks=all

Leader waits until all In-Sync Replicas (ISR) have replicated the message.

Safest option.

Interview Recommendation

Production systems usually use:

acks=all

along with idempotence enabled.

The outcome depends on when the crash occurs.

Case 1

Producer crashes before sending.

Result:

Message is lost because it never reached Kafka.

Case 2

Producer crashes after sending but before receiving ACK.

Producer cannot determine whether the broker stored the message.

If retries are enabled:

• Duplicate messages are possible without idempotence.
• With idempotence enabled, duplicates are prevented.

Case 3

Producer crashes after ACK.

The message is already stored safely in Kafka.

Interview Tip: Kafka guarantees message durability only after the broker successfully stores the record according to the configured acknowledgment level.

If sending fails because of a temporary issue (network failure, broker restart, leader election, etc.), the producer automatically retries.

Example

retries=5

Kafka retries the send up to five times.

Retry Flow

Producer

↓

Broker Failure

↓

Retry

↓

Broker Available

↓

Success

Important

Retries without idempotence can create duplicate messages.

Production configuration:

enable.idempotence=true

An idempotent producer ensures that even if the producer retries, Kafka stores only one copy of the message.

Enable it using:

enable.idempotence=true

Kafka assigns sequence numbers to messages from each producer and detects duplicate retries.

Benefits

• No duplicate messages due to retries
• Exactly-once write guarantee from the producer to Kafka
• Safer production systems

Interview One-liner

u003e An idempotent producer guarantees that retries do not result in duplicate messages in Kafka.

Instead of sending every message individually, Kafka groups multiple records into a batch before sending them.

Example

Without batching

Message1

↓

Network

Message2

↓

Network

Message3

↓

Network

With batching

Message1

Message2

Message3

↓

Single Network Request

Benefits

• Fewer network calls
• Better throughput
• Lower CPU usage
• Improved compression

Interview Tip: Batching is one of the major reasons Kafka can achieve millions of messages per second.

linger.ms specifies how long the producer waits before sending a batch.

Example

linger.ms=10

The producer waits up to 10 milliseconds to accumulate more messages before sending the batch.

Scenario

Without linger

Message

↓

Immediately Sent

With linger

Message

↓

Wait 10 ms

↓

More Messages Arrive

↓

Send Batch

Effect

Interview Tip: A small non-zero value (for example, 5–20 ms) often improves throughput significantly without noticeably increasing latency.

batch.size defines the maximum size (in bytes) of a batch before it is sent.

Example

batch.size=16384

Default is 16 KB.

If the accumulated batch reaches this size, Kafka sends it immediately, even if linger.ms has not expired.

Larger Batch

Advantages

• Better throughput
• Better compression
• Fewer network requests

Disadvantages

• More memory usage
• Slightly higher latency while batching

Relationship with linger.ms

A batch is sent when either:

• batch.size is reached, or
• linger.ms expires,

whichever happens first.

Kafka can compress batches before sending them to reduce network bandwidth and storage usage.

Configuration

compression.type=lz4

Supported algorithms:

Benefits

• Lower network bandwidth
• Reduced disk usage
• Higher throughput
• Faster replication for large workloads

Interview Recommendation

For most modern production systems:

• zstd offers the best balance of compression ratio and speed.
• lz4 is an excellent choice when very low latency is the primary goal.

@KafkaListener is a Spring Kafka annotation used to create a Kafka consumer.

Whenever a new message arrives in the subscribed topic, Spring automatically invokes the annotated method.

Instead of continuously writing polling logic, Spring Boot manages the consumer lifecycle.

Example

@Service
public class OrderConsumer {

    @KafkaListener(
            topics = "order-topic",
            groupId = "inventory-group")
    public void consume(OrderEvent order) {

        System.out.println(order);

    }

}

Internal Flow

Kafka Topic

↓

Kafka Consumer

↓

@KafkaListener

↓

Business Logic

Advantages

• Automatic polling
• Automatic deserialization
• Retry support
• Error handling
• Concurrent consumers
• Offset management

Interview Tip: @KafkaListener is built on top of KafkaConsumer API and continuously polls messages in the background.

Unlike producers, consumers pull messages from Kafka.

Kafka never pushes messages.

Consumer Flow

Consumer

↓

poll()

↓

Broker

↓

Messages Returned

↓

Business Logic

↓

Commit Offset

↓

poll() again

Steps

1. Consumer joins a Consumer Group.

2. Kafka assigns partitions.

3. Consumer continuously calls poll().

4. Broker returns available records.

5. Consumer processes records.

6. Offset is committed.

7. Consumer repeats polling.

Interview Tip: Kafka consumers are pull-based, which gives them control over consumption speed and supports back-pressure.

A Consumer Group is a collection of consumers working together to consume a topic.

Kafka distributes partitions among consumers in the same group.

Example

Topic

Order Topic

P0

P1

P2

Consumer Group

Inventory Group

Consumer A

Consumer B

Consumer C

Assignment

P0 → Consumer A

P1 → Consumer B

P2 → Consumer C

Rule

Inside one Consumer Group,

u003e One partition can be consumed by only one consumer at a time.

Benefits

• Load balancing
• Parallel processing
• High throughput
• Fault tolerance

Interview One-liner

A Consumer Group allows multiple consumers to share the work of consuming a topic while ensuring each message is processed only once within that group.

Without Consumer Groups

Consumer A

↓

Reads Everything

Only one consumer processes all messages.

With Consumer Groups

Consumer A → Partition 0

Consumer B → Partition 1

Consumer C → Partition 2

Benefits

• Horizontal scaling
• Better CPU utilization
• Faster processing
• Automatic failover

Real-world Example

Order Service produces

10000 orders/sec

One consumer cannot process them.

Instead

Inventory Group

Consumer 1

Consumer 2

Consumer 3

Consumer 4

Each consumer handles a subset of partitions.

One Consumer

Topic

↓

Consumer

Advantages

• Simple
• Easy ordering

Disadvantages

• Slow
• Single point of failure

Multiple Consumers

Topic

↓

Consumer A

Consumer B

Consumer C

Advantages

• Parallel processing
• Faster
• Fault tolerant
• Scalable

Interview Tip: The maximum parallelism is limited by the number of partitions, not by the number of consumers.

Suppose

Order Topic

Three different services consume the same topic.

Inventory Group

↓

Inventory Service

Payment Group

↓

Payment Service

Notification Group

↓

Notification Service

Every group receives all events independently.

Interview Tip: Each Consumer Group maintains its own offsets.

poll() is the most important API in Kafka Consumer.

It requests messages from the broker.

Example

ConsumerRecordsu003cString, Stringu003e records =
        consumer.poll(Duration.ofMillis(100));

What poll() does

• Fetches records
• Sends heartbeat
• Detects rebalance
• Returns available messages

Internal Flow

poll()

↓

Broker

↓

Records

↓

Business Logic

Interview Tip: If poll() is not called regularly, Kafka assumes the consumer has failed.

Consumer sends

poll()

Broker checks

New Messages?

If yes

Return Records

Otherwise

Consumer waits until:

• New data arrives
• Timeout occurs

Fetch Flow

Consumer

↓

poll()

↓

Broker

↓

Segment File

↓

Records Returned

It limits the maximum number of records returned in one poll.

Example

max.poll.records=500

Even if Kafka has

5000 messages

Consumer receives

500

at one time.

Why?

Prevents

• Huge memory usage
• Long processing time
• Consumer timeout

Interview Tip: Large values improve throughput but increase processing time per poll.

Consumer waits until at least this amount of data is available.

Example

fetch.min.bytes=1024

Kafka waits until

1 KB

of data is available before responding.

Benefit

Fewer network requests.

Better throughput.

Drawback

Higher latency.

Maximum time Kafka waits before responding to a fetch request.

Example

fetch.max.wait.ms=500

If fetch.min.bytes isn’t reached,

Kafka waits

500 ms

then sends available data.

Relationship

fetch.min.bytes

+

fetch.max.wait.ms

Together they balance

• Throughput
• Latency

Suppose

Consumer Group

Consumer A

Consumer B

Consumer C

Consumer B crashes.

Kafka detects

Missing heartbeat

↓

Triggers Rebalancing

↓

Partitions of Consumer B

↓

Assigned to A and C

Result

No message loss

Processing continues

Interview Tip: This automatic reassignment is one of Kafka’s major fault-tolerance features.

Rebalancing is the process of redistributing partitions among consumers in a Consumer Group.

Happens when

• New consumer joins
• Consumer leaves
• Consumer crashes
• Number of partitions changes

Example

Before

Consumer A

P0

P1

Consumer B

P2

P3

New Consumer C joins

After

Consumer A → P0

Consumer B → P1

Consumer C → P2,P3

Disadvantage

During rebalance

• Consumers temporarily stop processing.
• Throughput decreases.

Interview Tip: Kafka’s cooperative rebalancing (available in newer clients) reduces disruption compared to eager rebalancing.

Dynamic Membership

Default behavior.

Consumers receive temporary IDs.

When restarted,

Kafka performs a full rebalance.

Static Membership

Consumer has a fixed identity.

group.instance.id=consumer-1

If restarted quickly,

Kafka avoids unnecessary rebalancing.

Comparison

Interview Tip: Static membership reduces downtime in production clusters where consumers restart frequently.

An Offset is a unique sequential number assigned to every message within a partition.

Example

Partition 0

Offset 0 → Order Created

Offset 1 → Order Paid

Offset 2 → Order Shipped

Offset 3 → Order Delivered

Offsets are unique only within a partition.

Interview Tip: Offsets identify a message’s position in a partition log.

Offsets help Kafka:

• Track processed messages
• Resume after restart
• Prevent duplicate processing
• Support replay
• Enable fault tolerance

Example

Consumer processed

Offset 100

After restart,

Kafka resumes from

Offset 101

instead of starting over.

Interview One-liner

Offsets allow consumers to know exactly where to continue reading after failures or restarts.

Auto Commit

enable.auto.commit=true

Manual Commit

enable.auto.commit=false

Interview Recommendation

Production applications handling payments, inventory, or financial data generally prefer manual offset commits.

Disable auto commit.

enable.auto.commit=false

Then commit after successful processing.

Native Kafka Consumer example:

consumer.commitSync();

or

consumer.commitAsync();

With Spring Kafka, manual acknowledgment is commonly used.

@KafkaListener(topics = "order-topic")
public void consume(OrderEvent event,
                    Acknowledgment ack) {

    // Business Logic

    ack.acknowledge();
}

Interview Tip: Always commit offsets after successful business processing.

Suppose

Processed

Offset 100

Consumer crashes

↓

Offset not committed

↓

Consumer restarts

↓

Reads Offset 100 again

Result

Message is processed again.

Interview Tip: This behavior supports at-least-once delivery but requires applications to handle duplicate processing safely.

commitSync() commits offsets synchronously.

Example

consumer.commitSync();

Characteristics

• Blocks until Kafka confirms the commit
• Reliable
• Slightly slower

Advantages

• Stronger consistency
• Easy error handling

Disadvantages

• Higher latency
• Reduced throughput

commitAsync() commits offsets asynchronously.

Example

consumer.commitAsync();

Characteristics

• Non-blocking
• Better throughput
• Does not wait for broker acknowledgment

Advantages

• Faster
• Lower latency

Disadvantages

• Commit failures require careful handling because retries can overwrite newer committed offsets if implemented incorrectly.

Comparison

Interview Tip: Some production applications use commitAsync() during normal processing and commitSync() during shutdown to reduce the chance of losing the latest offset commit.

Kafka allows replaying events because messages remain in the log until retention expires.

Ways to reprocess:

• Reset consumer group offsets.
• Create a new Consumer Group (starts with fresh offsets).
• Seek to an earlier offset using the Consumer API.

Native API example:

consumer.seek(topicPartition, 0);

Real-world Use Cases

• Bug fixes
• Data migration
• Analytics rebuild
• Cache reconstruction

Configure:

auto.offset.reset=earliest

If no committed offsets exist for the consumer group, Kafka starts reading from the earliest retained message.

Interview Tip: auto.offset.reset is used only when there is no committed offset for the consumer group (or the committed offset is no longer valid).

Example

Topic contains

Offset 0

Offset 1

Offset 2

Offset 3

New Consumer Group starts:

earliest

Reads

0

1

2

3

latest

Ignores

0

1

2

3

Reads only future messages.

In At-most-once delivery, a message is delivered zero or one time.

A message may be lost, but it is never delivered twice.

Flow

Producer

↓

Kafka

↓

Consumer

↓

Commit Offset

↓

Process Message

Suppose

• Offset is committed first.
• Consumer crashes before processing.

Result

Message is permanently lost.

Characteristics

• No duplicate messages
• Possible message loss
• Highest performance
• Lowest reliability

Use Cases

• Application logs
• Monitoring metrics
• Analytics events
• Click tracking

Interview Tip: If losing a few messages is acceptable, At-most-once is sufficient.

In At-least-once delivery, every message is processed at least once.

No messages are lost.

However, duplicate processing is possible.

Flow

Consumer

↓

Process Message

↓

Commit Offset

Suppose

Consumer processes

Payment Successful

Before committing offset,

Consumer crashes.

After restart,

Kafka delivers the same message again.

Result

• No message loss
• Duplicate processing possible

Characteristics

• Reliable
• Possible duplicates
• Most common delivery guarantee

Use Cases

• Inventory
• Orders
• Email
• Payment Events (with idempotent business logic)

Interview Tip: Kafka provides At-least-once by default when offsets are committed after successful processing.

Exactly-once means

Every message is processed

Exactly one time

No duplicates.

No message loss.

Example

Transfer ₹1000

Without Exactly Once

Transfer ₹1000

↓

Consumer crashes

↓

Transfer ₹1000 again

Customer loses ₹2000.

With Exactly Once

Transfer ₹1000

↓

Processed Once

↓

Never Again

Interview Tip: Exactly-once is the strongest delivery guarantee but also the most complex.

Kafka provides Exactly Once Semantics (EOS) using multiple mechanisms together.

Step 1

Idempotent Producer

↓

No duplicate writes

Step 2

Transactional Producer

↓

Multiple writes become atomic

Step 3

Consumer reads only committed data

isolation.level=read_committed

Overall Flow

Producer

↓

Idempotence

↓

Transaction

↓

Kafka

↓

Committed

↓

Consumer

Interview Tip: Exactly-once is not achieved by one configuration.

It requires

• Idempotent Producer
• Transactions
• Consumer reading committed records
• Correct application design

Idempotence means

Repeating the same operation multiple times produces the same result.

Kafka’s Idempotent Producer prevents duplicate writes caused by retries.

Enable

enable.idempotence=true

Example

Without Idempotence

Producer sends

Order Created

Network failure

↓

Producer retries

↓

Kafka stores

Order Created

Order Created

Duplicate.

With Idempotence

Kafka detects duplicate sequence numbers.

Stores

Order Created

Only once.

Interview Tip: Idempotence prevents duplicates between Producer and Kafka Broker.

It does not prevent duplicate business processing after the message has already been consumed.

A Transactional Producer groups multiple Kafka operations into a single atomic transaction.

Either

Everything succeeds

OR

Everything rolls back.

Example

Need to write

Debit Topic

Credit Topic

Audit Topic

If one write fails,

No topic receives partial data.

Spring Boot Example

@Transactional
public void transfer() {

    kafkaTemplate.send("debit-topic", debit);

    kafkaTemplate.send("credit-topic", credit);

}

*(For Kafka transactions, the producer must also be configured with a transactional.id, and Kafka transactions are managed by the Kafka transaction manager.)*

Producer Configuration

spring.kafka.producer.transaction-id-prefix=tx-

Interview Tip: Kafka Transactions guarantee atomic writes across multiple partitions and topics.

Yes.

Kafka can lose messages if it is configured incorrectly.

Possible Reasons

acks=0

Producer never waits for broker confirmation.

Broker crashes.

Message lost.

acks=1

Leader crashes

before followers replicate.

Message lost.

Replication Factor = 1

Only one copy exists.

Broker dies.

Message lost.

Improper Offset Commit

Consumer commits

before processing.

Crash occurs.

Message lost from application’s perspective.

Prevent Data Loss

Use

acks=all

enable.idempotence=true

min.insync.replicas=2

replication.factor=3

Interview Tip: Kafka is designed to be highly durable, but durability depends on correct producer, broker, and consumer configuration.

Yes.

Duplicates are possible.

Scenario

Producer sends

↓

Broker stores

↓

ACK lost

↓

Producer retries

↓

Duplicate stored

Or

Consumer processes message

↓

Crash

↓

Offset not committed

↓

Kafka sends same message again

Prevent Duplicates

• Idempotent Producer
• Transactions
• Idempotent Consumer (business logic)
• Unique Business ID

Interview Tip: Duplicates are more common on the consumer side than on the producer side, so production applications often make consumers idempotent.

Kafka transmits data over the network as byte arrays.

Java Objects

OrderEvent

cannot be sent directly.

They must first be converted into bytes.

This process is called Serialization.

On the consumer side,

Bytes

↓

Java Object

This is called Deserialization.

Flow

Java Object

↓

Serializer

↓

Bytes

↓

Kafka

↓

Deserializer

↓

Java Object

Interview Tip: Kafka itself understands only byte arrays (byte[]).

StringSerializer

Converts String into bytes.

Example

kafkaTemplate.send("topic", "Hello");

Configuration

spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer

JsonSerializer

Converts Java Objects into JSON.

Example

OrderEvent order = new OrderEvent();

↓

{
  "id":101,
  "status":"CREATED"
}

Configuration

spring.kafka.producer.value-serializer=org.springframework.kafka.support.serializer.JsonSerializer

Comparison

Interview Tip: Use StringSerializer for plain text and JsonSerializer when sending DTOs or domain objects.

JsonDeserializer converts JSON back into Java Objects.

Producer

{
"id":101,
"status":"CREATED"
}

Consumer receives

OrderEvent

Configuration

spring.kafka.consumer.value-deserializer=org.springframework.kafka.support.serializer.JsonDeserializer

Spring Example

@KafkaListener(topics="order-topic")
public void consume(OrderEvent order){

}

Spring automatically converts JSON into OrderEvent.

Interview Tip: The producer and consumer must agree on the object structure, otherwise deserialization can fail.

Example

JSON

{
"id":101,
"name":"Laptop"
}

Avro

Binary Data

Much smaller.

Interview Tip: Avro is optimized for distributed systems where bandwidth, storage, and schema evolution matter.

Interview Recommendation

• Kafka ecosystems commonly use Avro with Schema Registry.
• gRPC-based microservices commonly use Protocol Buffers (Protobuf).

Because Avro offers:

• Small binary messages
• Faster serialization/deserialization
• Schema evolution
• Strong backward compatibility support
• Reduced storage usage
• Reduced network bandwidth

Real-world Benefits

Large organizations may process millions of events every minute.

Using Avro instead of JSON:

• Reduces network traffic
• Saves disk space
• Improves throughput
• Makes schema changes safer

Interview Tip: Avro is preferred in enterprise Kafka deployments because it combines efficient binary encoding with controlled schema evolution.

Schema Registry is a centralized service that stores and manages schemas used by producers and consumers.

Instead of embedding the full schema in every message, Kafka messages typically contain a schema identifier, and applications retrieve the schema from the registry.

Flow

Producer

↓

Schema Registry

↓

Kafka

↓

Consumer

↓

Schema Registry

Benefits

• Central schema management
• Version control
• Compatibility checks
• Prevents incompatible schema changes

Interview Tip: Schema Registry helps producers and consumers evolve independently while sharing compatible data contracts.

A new consumer can read data produced with an older schema.

Example

Version 1

id

name

Version 2

id

name

email

If email is optional or has a default value, new consumers can still read old messages.

Interview Tip: Backward compatibility is the most commonly used compatibility mode in Kafka environments.

An older consumer can read data produced with a newer schema.

Example

Older application

↓

Reads

New message

↓

Ignores newly added optional fields.

Interview Tip: Forward compatibility is useful when some services are upgraded later than others.

Full compatibility combines backward and forward compatibility.

This means:

• New consumers can read old data.
• Old consumers can read new data (provided schema changes remain compatible).

Comparison

Interview Tip: Schema Registry can enforce compatibility rules during schema registration, preventing incompatible schema versions from being published.

Spring Boot provides auto-configuration for Kafka through the Spring Kafka library.

application.properties

spring.kafka.bootstrap-servers=localhost:9092

# Producer
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.springframework.kafka.support.serializer.JsonSerializer

# Consumer
spring.kafka.consumer.group-id=order-group
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.springframework.kafka.support.serializer.JsonDeserializer

spring.kafka.consumer.properties.spring.json.trusted.packages=*

Interview Tip: At minimum, Kafka requires:

• Bootstrap Servers
• Producer Serializer
• Consumer Deserializer
• Consumer Group ID

Maven

u003cdependencyu003e
    u003cgroupIdu003eorg.springframework.kafkau003c/groupIdu003e
    u003cartifactIdu003espring-kafkau003c/artifactIdu003e
u003c/dependencyu003e

What it provides

• KafkaTemplate
• @KafkaListener
• Retry support
• Error handlers
• Transactions
• Batch listeners
• Dead Letter Topic support

Interview Tip: Spring Kafka is a wrapper around the native Kafka Java Client.

Producer

@Service
public class OrderProducer {

    @Autowired
    private KafkaTemplateu003cString, OrderEventu003e kafkaTemplate;

    public void publish(OrderEvent event){

        kafkaTemplate.send("order-topic", event);

    }

}

Send with Key

kafkaTemplate.send(
        "order-topic",
        event.getOrderId(),
        event
);

Interview Tip: Use business IDs as keys to preserve ordering.

@Service
public class InventoryConsumer {

    @KafkaListener(
            topics = "order-topic",
            groupId = "inventory-group")
    public void consume(OrderEvent order){

        System.out.println(order);

    }

}

Spring internally

Kafka

↓

Consumer

↓

@KafkaListener

↓

Business Logic

Interview Tip: Spring automatically manages polling, deserialization, thread management, and listener invocation.

DTO

public class OrderEvent {

    private Long orderId;

    private String status;

}

Producer

kafkaTemplate.send("order-topic", orderEvent);

Producer Configuration

spring.kafka.producer.value-serializer=org.springframework.kafka.support.serializer.JsonSerializer

What happens internally?

Java Object

↓

JSON

↓

Bytes

↓

Kafka

Interview Tip: Spring automatically converts Java objects into JSON using JsonSerializer.

Consumer Configuration

spring.kafka.consumer.value-deserializer=org.springframework.kafka.support.serializer.JsonDeserializer

Listener

@KafkaListener(topics="order-topic")
public void consume(OrderEvent order){

}

Internal Flow

Kafka Bytes

↓

JSON

↓

OrderEvent

Interview Tip: Producer and consumer must use compatible object structures.

Suppose

One Topic

order-topic

Inventory

@KafkaListener(topics="order-topic",
groupId="inventory-group")

Payment

@KafkaListener(topics="order-topic",
groupId="payment-group")

Notification

@KafkaListener(topics="order-topic",
groupId="notification-group")

Every consumer group receives all messages independently.

Interview Tip: Each Consumer Group maintains its own offsets.

If message processing fails,

Spring Kafka invokes an ErrorHandler.

Example

@Bean
public DefaultErrorHandler errorHandler(){

    return new DefaultErrorHandler();

}

What can ErrorHandler do?

• Retry
• Skip message
• Send to DLT
• Log exception

Interview Tip: DefaultErrorHandler replaced the older SeekToCurrentErrorHandler in modern Spring Kafka versions.

Spring Boot provides automatic retries.

Example

@RetryableTopic(attempts = "3")
@KafkaListener(topics="order-topic")
public void consume(OrderEvent order){

}

Internal Flow

Message

↓

Failure

↓

Retry 1

↓

Retry 2

↓

Retry 3

↓

Dead Letter Topic

Benefits

• Automatic retries
• Retry topics created automatically
• Delay support
• DLT support

Interview Tip: @RetryableTopic is much simpler than implementing retry logic manually.

What is DLT?

Messages that cannot be processed successfully after retries are sent to another topic called the Dead Letter Topic.

Flow

Order Topic

↓

Consumer

↓

Exception

↓

Retry

↓

Retry

↓

Dead Letter Topic

Spring Example

@RetryableTopic(attempts="3")
@KafkaListener(topics="order-topic")
public void consume(OrderEvent event){

    throw new RuntimeException();

}

@DltHandler
public void deadLetter(OrderEvent event){

    System.out.println(event);

}

Why DLT?

• Prevents blocking the main topic.
• Allows failed messages to be inspected and reprocessed.
• Improves system resilience.

Interview Tip: Never discard failed messages directly in production. Route them to a DLT.

A ConsumerFactory creates KafkaConsumer instances.

Example

@Bean
public ConsumerFactoryu003cString, OrderEventu003e consumerFactory(){

    Mapu003cString,Objectu003e config = new HashMapu003cu003e();

    config.put(
        ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,
        "localhost:9092");

    return new DefaultKafkaConsumerFactoryu003cu003e(config);

}

Why?

• Multiple consumer configurations
• Security (SSL/SASL)
• Custom deserializers
• Enterprise customization

Interview Tip: Spring Boot auto-configures a ConsumerFactory by default, but custom factories are useful for advanced requirements.

@Bean
public ProducerFactoryu003cString,OrderEventu003e producerFactory(){

    Mapu003cString,Objectu003e config = new HashMapu003cu003e();

    config.put(
            ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,
            "localhost:9092");

    return new DefaultKafkaProducerFactoryu003cu003e(config);

}

Use Cases

• Multiple clusters
• Different serializers
• Transactions
• Security
• Custom producer settings

This factory creates listener containers.

Example

@Bean
public ConcurrentKafkaListenerContainerFactoryu003cString,OrderEventu003e
listenerFactory(){

    ConcurrentKafkaListenerContainerFactoryu003cString,OrderEventu003e factory =
            new ConcurrentKafkaListenerContainerFactoryu003cu003e();

    factory.setConsumerFactory(consumerFactory());

    factory.setConcurrency(3);

    return factory;

}

What does Concurrency do?

Suppose

Topic

P0

P1

P2

Concurrency

3

Spring creates

Consumer1

Consumer2

Consumer3

Each processes one partition.

Interview Tip: Increasing concurrency beyond the number of partitions provides no additional parallelism.

Yes, but only within a single partition.

Kafka guarantees that messages written to the same partition are read in the exact order they were written.

Example

Partition 0

Offset 0 → Order Created

Offset 1 → Payment Completed

Offset 2 → Order Shipped

Offset 3 → Order Delivered

Consumer receives them in the same order.

Interview Tip: Kafka guarantees ordering per partition, not across the entire topic.

Within one partition

P0

Message1

Message2

Message3

Message4

Offsets

0

1

2

3

Consumer reads

0

↓

1

↓

2

↓

3

Order is always preserved.

Why?

Kafka stores data in an append-only log.

Interview Tip: Ordering is guaranteed because records are appended sequentially to a partition log.

Suppose

Order Topic

P0

P1

Messages

Order1

Order2

Order3

Order4

Distribution

P0

Order1

Order3

P1

Order2

Order4

Consumers process partitions independently.

Final execution may be

Order2

Order1

Order4

Order3

Interview Tip: Kafka does not guarantee global ordering across partitions.

Use the same message key for related events.

Example

kafkaTemplate.send(
        "order-topic",
        order.getOrderId(),
        order);

Partition calculation

hash(OrderId)

↓

Partition 1

All events for that order go to the same partition.

Example

Order 101

Created

Paid

Packed

Shipped

Delivered

All events remain in order.

Best Practices

• Use business IDs as keys.
• Keep related events in the same partition.
• Avoid changing partition counts unnecessarily for key-based workloads.

Message Keys determine the partition for a record.

Formula

hash(key) % partitionCount

Benefits

• Maintains ordering for related events.
• Ensures all events for the same entity reach the same partition.
• Improves data locality.

Real-world Examples

Use:

• Order ID
• Customer ID
• Account ID
• Product ID

Avoid random UUIDs if ordering by business entity is required.

Interview Tip: The message key is the primary mechanism for preserving event ordering in Kafka.

Yes.

Ordering can be affected in several situations.

Multiple Partitions

Related events sent with different keys can land in different partitions.

Producer Retries

If idempotence is disabled and retries are misconfigured, ordering can be affected in certain failure scenarios.

Multiple Producers

If multiple producers publish updates for the same business entity without a consistent partitioning strategy, event order can become unpredictable.

Increasing Partitions

Changing the number of partitions changes the result of:

hash(key) % partitionCount

Future events for the same key may be routed to a different partition, although existing records remain in their original partitions.

Interview Tip: Ordering is easiest to preserve by using a stable business key and keeping the partition count stable when possible.

Suppose

Topic

P0

P1

P2

Consumer Group

Consumer A

Consumer B

Consumer C

Assignment

P0 → A

P1 → B

P2 → C

Each consumer processes its assigned partition sequentially.

Important Rule

Ordering is preserved within each assigned partition.

Parallel processing occurs across different partitions.

Example

P0

A

B

C

Always processed as

A

↓

B

↓

C

while another partition is processed independently.

A Partition is a physical division of a Kafka Topic.

Instead of storing all messages in one large log, Kafka divides them into multiple partitions.

Without partitioning

Order Topic

↓

Single Log File

With partitioning

Order Topic

↓

Partition 0

Partition 1

Partition 2

Why is it needed?

Partitioning provides:

• Parallel processing
• Horizontal scalability
• Higher throughput
• Better load balancing
• Fault tolerance through replication

Example

Suppose

10000 Orders/sec

One partition

↓

One consumer

↓

Slow processing

With four partitions

P0

P1

P2

P3

Four consumers can process data simultaneously.

Interview Tip: Kafka scales primarily by adding partitions, not by increasing machine size.

There is no fixed number.

It depends on:

• Expected throughput
• Number of consumers
• Future scalability
• Ordering requirements
• Broker capacity

Example

Suppose

Need

20000 Messages/sec

Processing capacity

One Consumer

↓

5000 Messages/sec

Required

4 Partitions

↓

4 Consumers

Best Practices

• Start with expected future traffic.
• Keep extra capacity for growth.
• Avoid creating thousands of partitions unnecessarily.
• Balance throughput with operational overhead.

Interview Tip: Maximum parallelism equals the number of partitions.

Yes.

Kafka allows increasing the partition count.

Example

Before

Order Topic

P0

P1

Increase

↓

After

P0

P1

P2

P3

Command

kafka-topics.sh --alter \\
--topic order-topic \\
--partitions 4

Interview Tip: Increasing partitions is supported without deleting the topic.

After increasing partitions,

New messages may go to the newly created partitions.

Old messages remain in their original partitions.

Example

Before

Order ID 101

↓

Partition 0

After increasing partitions,

Future events for the same key may hash differently.

hash(key) % partitionCount

If partition count changes,

Partition mapping changes.

Important

Ordering for future events of the same key may no longer continue in the original partition.

Existing data is not redistributed automatically.

Interview Tip: Increasing partitions can affect key-based ordering because the hash calculation depends on the partition count.

Kafka balances load through partition assignment.

Producer

↓

Partitions

↓

Consumers

Example

Topic

P0

P1

P2

P3

Consumer Group

Consumer A

Consumer B

Consumer C

Consumer D

Assignment

P0 → A

P1 → B

P2 → C

P3 → D

Each consumer processes a different partition.

Producer Side

If no key is present,

Kafka distributes records using its default partitioner (modern clients typically use sticky partitioning for better batching efficiency).

If a key exists,

hash(key)

↓

Partition

Interview Tip: Load balancing occurs on both the producer side (partition selection) and the consumer side (partition assignment within a consumer group).

No.

Kafka does not support decreasing the number of partitions.

Suppose

P0

P1

P2

P3

Cannot become

P0

P1

Why?

Because

• Existing offsets would become invalid.
• Data movement would be complex.
• Ordering could break.
• Consumer assignments would be affected.

If fewer partitions are needed

Create a new topic

↓

Copy data

↓

Switch applications

Interview Tip: Kafka supports increasing partitions, but not decreasing them.

A Hot Partition occurs when one partition receives much more traffic than the others.

Example

Customer A

↓

100000 Messages

All routed to

Partition 0

Other partitions remain mostly idle.

Result

P0

100000 Messages

P1

100 Messages

P2

50 Messages

Problems

• High CPU usage
• High disk I/O
• Consumer lag
• Slow processing

Solutions

• Choose a better partition key.
• Increase key cardinality.
• Split large workloads.
• Review business design.

Interview Tip: Hot partitions are usually caused by poor key selection rather than Kafka itself.

The partition key decides where messages are stored.

Good keys:

• Order ID
• Customer ID
• Account ID
• Product ID

Poor keys:

Country

↓

INDIA

Millions of messages

↓

One partition

Hot Partition

Good Example

Order ID

101

102

103

104

Messages distribute naturally while preserving order for each order.

Best Practices

• High cardinality
• Even distribution
• Stable value
• Preserve business ordering

Interview Tip: The partition key should balance ordering requirements and load distribution.

Replication Factor defines how many copies of each partition Kafka maintains.

Example

Replication Factor = 3

Partition 0

Leader → Broker 1

Follower → Broker 2

Follower → Broker 3

Three copies exist.

Benefits

• High availability
• Fault tolerance
• Data durability

Interview Tip: Replication Factor must not exceed the number of brokers in the cluster.

Without replication

Broker

↓

Crash

↓

Data Lost

With replication

Leader

↓

Followers

↓

Leader crashes

↓

Follower becomes Leader

Advantages

• Prevents data loss
• High availability
• Automatic recovery
• Disaster resilience

Interview Tip: Replication protects Kafka from broker failures.

Suppose

Broker 1

Leader

↓

Crash

Kafka Controller detects the failure.

↓

Checks ISR.

↓

Promotes an in-sync follower.

↓

Producer and Consumer automatically communicate with the new leader.

Result

System continues with minimal interruption.

Interview Tip: If an ISR replica is available, applications usually continue without manual intervention.

Every partition has

• One Leader
• Multiple Followers

Example

Leader

Broker 1

Followers

Broker 2

Broker 3

Leader crashes

↓

Kafka Controller

↓

Selects new Leader from ISR

↓

Clients update metadata

↓

Continue processing

Interview Tip: Leader election is performed independently for each partition.

Kafka assigns a preferred leader for each partition, typically the first replica in the replica assignment.

Example

Broker 1

Preferred Leader

Broker 2

Follower

Broker 3

Follower

After failures and recovery,

Kafka can move leadership back to the preferred leader to rebalance leadership across brokers.

Benefits

• Balanced leadership
• Better resource utilization
• Predictable cluster behavior

Interview Tip: The preferred leader is not always the current leader after failures.

Normally,

Only ISR members can become leaders.

If ISR becomes unavailable,

Kafka can optionally elect an out-of-sync replica.

This is called Unclean Leader Election.

Example

Leader

↓

Crash

↓

ISR Empty

↓

Out-of-sync Follower

↓

Becomes Leader

Advantage

Cluster becomes available sooner.

Disadvantage

Messages that existed only on the failed leader may be lost.

Production Recommendation

unclean.leader.election.enable=false

Interview Tip: Most production systems disable unclean leader election to prioritize data consistency over availability.

min.insync.replicas defines the minimum number of in-sync replicas (including the leader) required to acknowledge a write when the producer uses:

acks=all

Example

Replication Factor = 3

Leader

Follower

Follower

Configuration

min.insync.replicas=2

At least

Leader

+

One Follower

must successfully replicate the message before the write is acknowledged.

If fewer than two ISR replicas are available, Kafka rejects the write with an error instead of silently risking data loss.

Interview Tip: acks=all works together with min.insync.replicas to improve durability.

Replica Lag is the delay between the leader and follower replicas.

Example

Leader

Offset 1000

Follower

Offset 995

Lag

5 Messages

Causes

• Slow network
• Heavy disk I/O
• High traffic
• Slow broker

Problems

• Increased failover risk
• Reduced ISR membership
• Higher replication latency

Interview Tip: Large replica lag can cause followers to leave the ISR if they fall too far behind.

Kafka achieves High Availability through several mechanisms working together.

Components

• Replication
• Multiple brokers
• Leader election
• ISR
• Automatic failover
• Producer acknowledgments
• Consumer Groups

Architecture

Producer

↓

Leader

↓

Followers

↓

Consumer Group

If a broker fails,

↓

Leader election occurs.

↓

Consumers continue processing.

Production Best Practices

acks=all

enable.idempotence=true

replication.factor=3

min.insync.replicas=2

Interview Tip: High Availability in Kafka is achieved through replication, ISR-based leader election, and automatic failover, allowing the cluster to continue serving producers and consumers even when brokers fail.

When a Kafka consumer throws an exception while processing a message, the message is not considered successfully processed.

What happens next depends on the consumer configuration.

Possible actions:

• Retry the message
• Skip the message
• Send it to a Dead Letter Topic (DLT)
• Stop the listener (depending on configuration)

Example

Kafka Topic

↓

Consumer

↓

Business Logic

↓

Exception

Without error handling

↓

Consumer repeatedly fails on the same message.

Spring Boot Example

@KafkaListener(topics = "order-topic")
public void consume(OrderEvent order) {

    if(order.getAmount() u003c 0){
        throw new IllegalArgumentException("Invalid Amount");
    }

}

Interview Tip: A failed message should not block the entire application. Use retries and a Dead Letter Topic for robust processing.

Retries allow Kafka consumers to process transient failures automatically.

Typical retry flow:

Message

↓

Processing

↓

Exception

↓

Retry

↓

Retry

↓

Retry

↓

Success

If retries are exhausted,

↓

Dead Letter Topic

Spring Boot Example

@RetryableTopic(attempts = "3")
@KafkaListener(topics = "order-topic")
public void consume(OrderEvent event){

}

Benefits

• Handles temporary failures
• Reduces manual intervention
• Improves system resilience

Interview Tip: Retry only transient failures such as temporary network or database issues.

Retryable Exceptions

Temporary problems.

Examples

• Database temporarily unavailable
• Network timeout
• External API timeout
• Broker temporarily unavailable

Retry usually succeeds.

Non-Retryable Exceptions

Permanent problems.

Examples

• Invalid JSON
• Invalid business data
• Missing mandatory fields
• IllegalArgumentException
• Validation failures

Retrying these messages usually produces the same failure.

Comparison

Interview Tip: Do not retry validation or business rule failures. Send them directly to the Dead Letter Topic.

A Dead Letter Queue stores messages that cannot be processed successfully.

Although “DLQ” is a generic messaging term, in Kafka this concept is usually implemented as a Dead Letter Topic (DLT).

Flow

Kafka

↓

Consumer

↓

Exception

↓

Retries

↓

DLQ / DLT

Benefits

• Prevents endless retries
• Prevents blocking the main topic
• Allows manual investigation
• Enables later reprocessing

Interview Tip: In Kafka, the implementation is typically a Dead Letter Topic, even though many interviewers may casually say “DLQ.”

A Dead Letter Topic is a Kafka topic that stores failed messages after all retry attempts have been exhausted.

Example

order-topic

↓

Consumer

↓

Failure

↓

Retry

↓

Retry

↓

order-topic.DLT

Spring Example

@RetryableTopic(attempts = "3")
@KafkaListener(topics = "order-topic")
public void consume(OrderEvent event){

    throw new RuntimeException();

}

@DltHandler
public void processDlt(OrderEvent event){

    System.out.println(event);

}

Benefits

• Failed messages are preserved.
• Business processing continues.
• Easier debugging.
• Safe reprocessing.

Interview Tip: Never discard failed production messages. Route them to a Dead Letter Topic.

Sometimes failures occur before the listener is invoked, during deserialization.

Example

Invalid JSON

↓

Deserializer throws exception

↓

Listener is never called

ErrorHandlingDeserializer catches these failures and passes error information to Spring Kafka.

Configuration

spring.kafka.consumer.key-deserializer=org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
spring.kafka.consumer.value-deserializer=org.springframework.kafka.support.serializer.ErrorHandlingDeserializer

spring.kafka.consumer.properties.spring.deserializer.key.delegate.class=org.apache.kafka.common.serialization.StringDeserializer

spring.kafka.consumer.properties.spring.deserializer.value.delegate.class=org.springframework.kafka.support.serializer.JsonDeserializer

Interview Tip: Use ErrorHandlingDeserializer when malformed messages should be handled gracefully instead of crashing the consumer.

SeekToCurrentErrorHandler was the error handler used in older versions of Spring Kafka.

Its behavior:

• Seeks the consumer back to the failed offset.
• Retries processing.
• Can eventually publish to a Dead Letter Topic.

Current Status

It has been deprecated and replaced by DefaultErrorHandler.

Interview Tip: If asked in modern Spring Boot interviews, mention that DefaultErrorHandler is the recommended replacement.

DefaultErrorHandler is the recommended error handler in modern Spring Kafka.

Example

@Bean
public DefaultErrorHandler errorHandler() {

    return new DefaultErrorHandler();

}

Features

• Retry support
• Backoff strategies
• Dead Letter Topic integration
• Exception classification
• Recovery callbacks

Example

Message

↓

Exception

↓

Retry

↓

Retry

↓

Dead Letter Topic

Interview Tip: Use DefaultErrorHandler for centralized retry and recovery handling.

Backoff determines how long Kafka waits before retrying a failed message.

Fixed Backoff

Waits a constant interval.

Retry

↓

5 sec

↓

Retry

↓

5 sec

↓

Retry

Example

new FixedBackOff(5000L, 3);

Exponential Backoff

Delay increases after every retry.

Retry

↓

1 sec

↓

2 sec

↓

4 sec

↓

8 sec

Benefits

• Reduces pressure on dependent systems.
• Gives external services time to recover.
• Prevents retry storms.

Interview Tip: Exponential backoff is generally preferred for external services because it avoids overwhelming a failing dependency.

Producer performance can be improved using batching, compression, asynchronous sending, and proper acknowledgment settings.

Important Configurations

acks=1
linger.ms=10
batch.size=65536
compression.type=zstd

Techniques

• Enable batching.
• Increase batch.size.
• Set a small non-zero linger.ms.
• Use zstd or lz4 compression.
• Send messages asynchronously.
• Use an appropriate number of partitions.

Interview Tip: The biggest producer performance gains usually come from batching and compression.

Consumer performance depends on how efficiently records are fetched and processed.

Important Configurations

max.poll.records=500
fetch.min.bytes=10240

Techniques

• Increase partitions for parallelism.
• Increase listener concurrency.
• Tune max.poll.records.
• Increase fetch.min.bytes for larger fetches.
• Optimize business logic.
• Avoid long-running processing inside the listener thread.

Interview Tip: Consumer performance is often limited more by slow business processing than by Kafka itself.

Defines the maximum size (in bytes) of a producer batch.

Example

batch.size=65536

(64 KB)

Larger Batch

Advantages

• Higher throughput
• Better compression
• Fewer network requests

Disadvantages

• Higher memory usage
• Slightly increased latency

Interview Tip: batch.size works together with linger.ms.

Defines how long the producer waits to accumulate more records before sending a batch.

Example

linger.ms=10

Producer waits up to 10 ms before sending.

Small Value

• Lower latency
• Smaller batches

Large Value

• Better throughput
• Larger batches

Interview Tip: A small non-zero value often provides a good balance between latency and throughput.

Compression reduces network bandwidth and storage requirements.

Example

compression.type=zstd

Comparison

Interview Tip: zstd is often the best default for modern Kafka deployments, while lz4 is preferred when minimizing latency.

Limits the maximum number of records returned in one poll() call.

Example

max.poll.records=500

Higher Value

Advantages

• Better throughput
• Fewer poll requests

Disadvantages

• Longer processing time
• Greater chance of exceeding processing limits if work is slow

Interview Tip: Choose a value that your application can process comfortably before the next poll.

Minimum amount of data the broker should collect before responding to a fetch request.

Example

fetch.min.bytes=10240

Benefits

• Larger fetches
• Fewer network calls
• Higher throughput

Drawback

Higher latency during periods of low traffic.

Interview Tip: Useful for high-volume consumers.

Defines the maximum amount of data the consumer can receive in a single fetch response.

Example

fetch.max.bytes=52428800

(50 MB)

Benefits

• Fewer fetch requests for large workloads
• Improved throughput

Drawback

Higher memory usage on the consumer.

Interview Tip: Tune this based on your average message size and available memory.

Recommended Production Configuration

acks=1
linger.ms=10
batch.size=65536
compression.type=zstd
enable.idempotence=true

Additional Best Practices

• Use asynchronous sends.
• Avoid unnecessary flush() calls.
• Increase partitions for greater write parallelism.
• Choose keys that distribute load evenly.

Interview Tip: If reliability is more important than raw throughput, use:

acks=all
enable.idempotence=true

even though it may reduce throughput slightly.

Recommended Configuration

max.poll.records=500
fetch.min.bytes=10240
fetch.max.bytes=52428800

Additional Best Practices

• Increase partition count to enable more parallel consumers.
• Configure ConcurrentKafkaListenerContainerFactory concurrency appropriately.
• Keep listener processing lightweight.
• Offload slow tasks to worker threads or downstream services when appropriate.
• Scale consumer instances horizontally when needed.

A Kafka Transaction allows a producer to send multiple messages as one atomic unit.

This means:

• Either all messages are committed
• Or none of them are visible to consumers using read_committed.

Without Transaction

Producer

↓

Topic A ✔

↓

Topic B ✖ (Failure)

Result

Topic A → Message Exists

Topic B → No Message

Data becomes inconsistent.

With Transaction

Producer

↓

Begin Transaction

↓

Topic A

↓

Topic B

↓

Commit Transaction

If any step fails

↓

Entire transaction is rolled back.

Interview Tip: Kafka Transactions guarantee atomic writes across multiple partitions and topics.

Transactions solve partial write problems.

Example

Order Processing

Save Payment Event

↓

Save Inventory Event

↓

Save Notification Event

Suppose

Payment ✔

Inventory ✔

Notification ✖

Business becomes inconsistent.

Using Transactions

Either

All 3 Events Commit

OR

None Commit

Benefits

• Atomic writes
• Exactly Once Processing
• Prevents partial updates
• Better consistency

Interview Tip: Transactions are useful when multiple related Kafka writes must succeed or fail together.

A Transactional Producer is a Kafka producer configured with a transactional ID.

Configuration

spring.kafka.producer.transaction-id-prefix=order-tx-

Or using the native client

Properties props = new Properties();

props.put(
    ProducerConfig.TRANSACTIONAL_ID_CONFIG,
    "order-service-1");

Initialize

producer.initTransactions();

Interview Tip: Every transactional producer must have a unique transactional.id.

Starts a new Kafka transaction.

Example

producer.beginTransaction();

Flow

Begin Transaction

↓

Send Event 1

↓

Send Event 2

↓

Commit

Until committed,

Consumers configured with read_committed cannot read the messages.

Interview Tip: Always call beginTransaction() before sending transactional records.

Commits all messages in the current transaction.

Example

producer.commitTransaction();

Flow

Begin

↓

Message A

↓

Message B

↓

Commit

After commit,

Messages become visible.

Interview Tip: A committed transaction makes all records atomically visible.

Cancels the current transaction.

Example

producer.abortTransaction();

Suppose

Message A Sent

↓

Message B Sent

↓

Database Error

Abort

↓

Neither message becomes visible to read_committed consumers.

Interview Tip: Use abortTransaction() whenever business processing fails before commit.

Example

Without Transactions

Order Topic ✔

Payment Topic ✖

With Transactions

Order Topic ✔

Payment Topic ✔

OR

Neither

Interview Tip: Idempotence prevents duplicate writes.

Transactions prevent partial writes.

Exactly Once Processing (EOS) is achieved using:

• Idempotent Producer
• Transactional Producer
• Consumer configured with

isolation.level=read_committed

Flow

Begin Transaction

↓

Read Event

↓

Business Logic

↓

Produce Event

↓

Commit Transaction

If processing fails

↓

Abort Transaction

↓

Consumer does not see partial output.

Production Configuration

enable.idempotence=true

transactional.id=order-service-1

acks=all

isolation.level=read_committed

Interview Tip: Transactions alone do not guarantee end-to-end exactly-once business processing. Your application logic must still be idempotent where necessary.

Kafka Streams is a Java library for building real-time stream processing applications.

It can:

• Read from Kafka topics
• Transform events
• Filter data
• Join streams
• Aggregate data
• Write results back to Kafka

Architecture

Input Topic

↓

Kafka Streams Application

↓

Output Topic

Example

Order Topic

↓

Calculate Daily Sales

↓

Sales Topic

Interview Tip: Kafka Streams is a client library, not a separate server or cluster.

Example

Consumer API

Consume

↓

Process

↓

Store State Yourself

Kafka Streams

Consume

↓

Aggregate

↓

Join

↓

Window

↓

Publish

Interview Tip: Use Consumer API for straightforward message consumption and Kafka Streams for continuous event processing and analytics.

KStream

Represents an immutable stream of events.

Example

Order Created

↓

Order Updated

↓

Order Delivered

Every event is preserved.

KTable

Represents the latest state for each key.

Example

Order 101

Created

↓

Paid

↓

Delivered

Only the current state for Order 101 is retained.

Comparison

Interview Tip: Think of a KStream as an event log and a KTable as a continuously updated table.

Stateless operations process each event independently.

Examples

• Filter
• Map
• FlatMap
• Peek

Example

Orders

↓

Filter Amount > ₹1000

↓

Output Topic

No previous events are required.

Interview Tip: Stateless processing does not maintain local state.

Stateful operations depend on previously processed data.

Examples

• Aggregations
• Counting
• Joins
• Windowing

Example

Orders

↓

Count Orders Per Customer

↓

State Store

↓

Result

Kafka Streams stores state locally and backs it up using Kafka changelog topics.

Interview Tip: Stateful operations use embedded state stores with fault-tolerant changelog topics.

Windowing groups events within a time interval.

Example

5-minute window

10:00

↓

10:05

↓

Count Orders

Instead of counting forever,

Kafka Streams counts only within that window.

Types

• Tumbling Window
• Hopping Window
• Sliding Window
• Session Window

Use Cases

• Orders per minute
• Clicks per hour
• Fraud detection
• Real-time dashboards

Interview Tip: Windowing enables time-based analysis of streaming data.

Kafka Streams can join multiple streams or tables.

Common joins

• KStream–KStream
• KStream–KTable
• KTable–KTable

Example

Orders

+

Customers

↓

Enriched Order

Result

Order

+

Customer Name

↓

Output Topic

Interview Tip: KStream–KTable joins are commonly used to enrich event streams with reference data.

Aggregations compute summary values over streams.

Examples

• Count
• Sum
• Average
• Max
• Min

Example

Orders

↓

Group By Customer

↓

Count

↓

Customer Order Count

Use Cases

• Daily sales
• Active users
• Inventory totals
• Order statistics

Interview Tip: Aggregations are typically combined with groupBy() and often with windowing for real-time analytics.

Kafka monitoring involves observing the health and performance of:

• Producers
• Consumers
• Brokers
• Topics
• Partitions
• Replication
• Disk usage
• JVM

A typical production monitoring architecture looks like this:

Kafka Cluster

↓

JMX Metrics

↓

Prometheus

↓

Grafana Dashboard

↓

Alerts

Important Metrics

• Consumer Lag
• Producer Throughput
• Broker CPU
• Disk Usage
• Request Latency
• ISR Count
• Under Replicated Partitions
• Offline Partitions

Interview Tip: Monitoring Kafka means monitoring the entire ecosystem, not just the brokers.

Consumer Lag is the difference between:

Latest Offset

and

Consumer Offset

Formula

Consumer Lag

=

Latest Offset

-

Committed Offset

Example

Latest Offset

1000

Consumer Offset

980

Lag

20 Messages

Causes

• Slow consumer
• Heavy processing
• Network issues
• Database latency
• Too few consumers

Problems

• Delayed processing
• High latency
• Memory pressure
• SLA violations

Solutions

• Increase partitions
• Add more consumers
• Optimize business logic
• Increase concurrency
• Scale consumer instances

Interview Tip: Consumer Lag is the most important production metric for Kafka consumers.

Important producer metrics include:

• Messages/sec
• Bytes/sec
• Record Send Rate
• Record Error Rate
• Retry Rate
• Request Latency
• Compression Ratio
• Batch Size
• Buffer Utilization

Example

Producer

↓

Kafka

↓

20000 Messages/sec

What to Monitor

• Failed sends
• High retries
• Large latency
• Low throughput

Interview Tip: A sudden increase in producer retries often indicates broker or network issues.

Broker metrics indicate the health of Kafka servers.

Important metrics:

• CPU Usage
• Memory Usage
• Disk Usage
• Network I/O
• Request Latency
• Active Controllers
• Under Replicated Partitions
• Offline Partitions
• ISR Shrink/Expand Events

Example

Broker

↓

CPU

Disk

Network

Memory

Interview Tip: Monitor Under Replicated Partitions (URP) because they indicate replication problems.

Several tools help visualize Kafka clusters.

Popular tools:

• AKHQ
• Kafka UI (Provectus)
• Conduktor
• CMAK (Cluster Manager for Apache Kafka)

Features

• Topic browsing
• Consumer Groups
• Consumer Lag
• Message inspection
• Partition details
• Broker status

Interview Tip: AKHQ and Kafka UI are among the most widely used open-source tools for day-to-day Kafka administration.

This is the most common production monitoring stack.

Architecture

Kafka

↓

JMX Exporter

↓

Prometheus

↓

Grafana

↓

Dashboard

Prometheus

Collects Kafka metrics.

Examples

• Consumer Lag
• CPU
• JVM Memory
• Network

Grafana

Visualizes

• Dashboards
• Alerts
• Trends

Typical Dashboard

Kafka Dashboard

Consumer Lag

Broker Health

Disk Usage

Producer Rate

JVM

Interview Tip: Prometheus collects metrics, while Grafana visualizes them.

Kafka logs help diagnose production issues.

Important logs include:

• Broker logs
• Consumer logs
• Producer logs
• Controller logs

Common Issues

• Leader Election
• Broker restart
• Replica lag
• Authentication failures
• Network timeout
• Serialization errors

Interview Tip: Always check application logs together with Kafka broker logs when investigating production failures.

A consumer is considered slow when it cannot process messages as quickly as they arrive.

Symptoms

Producer

10000/sec

↓

Consumer

3000/sec

Lag continues increasing.

Detection

• Increasing Consumer Lag
• High Processing Time
• Growing Queue Size
• Frequent Rebalances

Solutions

• Add consumers
• Increase partitions
• Optimize processing
• Tune max.poll.records
• Scale horizontally

Interview Tip: Consumer Lag is the primary indicator of a slow consumer.

Kafka stores messages on disk.

If disks become full,

Kafka may stop accepting new writes.

Monitor

• Used Space
• Free Space
• Log Directory Usage
• Retention Cleanup
• Segment Growth

Example

Disk

100 GB

↓

95 GB Used

Critical Alert.

Solutions

• Increase storage
• Reduce retention
• Enable log cleanup
• Add brokers

Interview Tip: Disk monitoring is essential because Kafka is a disk-based messaging system.

Topics continuously grow as new messages arrive.

Monitor:

• Topic Size
• Partition Size
• Message Count
• Growth Rate

Example

order-topic

↓

400 GB

Large topics increase:

• Recovery time
• Replication time
• Disk usage

Interview Tip: Monitor topic growth trends instead of waiting until disks become full.

Partition imbalance occurs when leaders or partitions are unevenly distributed across brokers.

Example

Broker 1

80 Partitions

Broker 2

10 Partitions

Broker 3

10 Partitions

Problems

• High CPU
• High Disk Usage
• Uneven Load
• Slow Processing

Solutions

• Reassign partitions
• Rebalance preferred leaders
• Review partition strategy

Interview Tip: Balanced partition leadership improves cluster performance and resource utilization.

Authentication verifies the identity of clients before they connect to Kafka.

Kafka supports:

• SSL Authentication
• SASL Authentication

Flow

Producer

↓

Authentication

↓

Kafka Broker

↓

Authorized Connection

Why?

Authentication prevents unauthorized clients from accessing the cluster.

Interview Tip: Authentication answers “Who are you?”

Authorization answers “What are you allowed to do?”

SSL (TLS) encrypts communication between:

• Producer ↔ Broker
• Consumer ↔ Broker
• Broker ↔ Broker

Flow

Producer

⇄ Encrypted SSL/TLS ⇄

Kafka Broker

Benefits

• Encryption
• Data integrity
• Optional client certificate authentication
• Protection against network eavesdropping

Common Configuration

security.protocol=SSL
ssl.truststore.location=/path/truststore.jks
ssl.keystore.location=/path/keystore.jks

Interview Tip: SSL primarily protects data in transit.

SASL (Simple Authentication and Security Layer) provides client authentication.

Common mechanisms:

• PLAIN
• SCRAM-SHA-256
• SCRAM-SHA-512
• GSSAPI (Kerberos)
• OAUTHBEARER

Flow

Producer

↓

Username / Credentials

↓

Kafka Broker

↓

Authentication Success

Configuration Example

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512

Interview Tip: Many enterprise deployments use SASL_SSL, combining authentication with encrypted communication.

ACLs determine which authenticated users can perform specific operations.

Permissions include:

• Read
• Write
• Create
• Delete
• Alter
• Describe

Example

User A

↓

Read

order-topic

User B

↓

Write

payment-topic

Benefits

• Fine-grained access control
• Least privilege
• Multi-team security

Interview Tip: ACLs implement authorization after authentication succeeds.

Topic authorization restricts which users or applications can access specific topics.

Example

Inventory Service

↓

Read

inventory-topic

Payment Service

↓

Write

payment-topic

Unauthorized requests are rejected.

Interview Tip: Use ACLs to implement topic-level authorization.

Kafka encrypts communication using SSL/TLS, but applications may also encrypt the message payload before sending it.

Example

Customer Data

↓

AES Encryption

↓

Kafka

↓

Consumer

↓

Decrypt

Why?

Provides end-to-end protection for highly sensitive information, even if someone gains access to Kafka storage.

Interview Tip: SSL encrypts data in transit; application-level encryption protects the payload itself.

A secure producer should:

• Use SSL/TLS
• Authenticate with SASL (or client certificates)
• Use strong serializers
• Enable idempotence
• Validate server certificates

Example

security.protocol=SASL_SSL
enable.idempotence=true
acks=all

Best Practices

• Store credentials securely.
• Avoid hardcoding secrets.
• Rotate certificates and passwords regularly.

Interview Tip: A secure producer combines authentication, encryption, and reliable delivery.

A secure consumer should:

• Authenticate using SASL or SSL client certificates.
• Use encrypted communication.
• Have only the required topic permissions.
• Validate broker certificates.
• Protect committed offsets and credentials.

Example

security.protocol=SASL_SSL
group.id=inventory-group

Best Practices

• Grant minimum required ACL permissions.
• Keep credentials outside source code.
• Monitor authentication failures.

Interview Tip: Security follows the principle of least privilege—consumers should have only the permissions they actually need.

Kafka is commonly used as an asynchronous communication layer between microservices.

Instead of services calling each other directly using REST, they communicate through events.

Without Kafka (Synchronous)

Order Service
      |
      +----u003e Payment Service
      |
      +----u003e Inventory Service
      |
      +----u003e Notification Service

Problems:

• Tight coupling
• High latency
• Cascading failures
• Difficult scaling

With Kafka (Event-Driven)

Order Service
      |
      | OrderCreated Event
      |
      ▼
   Kafka Topic
      |
-------------------------------
|         |          |         |
Payment Inventory Notification Analytics

Benefits

• Loose coupling
• Independent deployments
• Better scalability
• Better fault tolerance
• Event replay
• Asynchronous processing

Interview Tip: Kafka acts as the communication backbone in an event-driven microservices architecture.

Event-Driven Architecture (EDA) is a design pattern where services communicate by publishing and consuming events instead of making synchronous API calls.

Example

Customer Places Order

↓

OrderCreated Event

↓

Kafka

↓

Payment

↓

Inventory

↓

Shipping

↓

Notification

Each service reacts independently.

Advantages

• Loose coupling
• High scalability
• Better resilience
• Independent deployment
• Easy addition of new services

Interview Tip: EDA improves scalability because producers don’t wait for consumers.

Instead of calling Payment Service using REST,

Order Service publishes

OrderCreated Event

Flow

Order Service

↓

Kafka

↓

Payment Service

↓

PaymentCompleted Event

↓

Kafka

↓

Order Service

Order Service updates the order status after consuming PaymentCompleted.

Why Kafka instead of REST?

• Payment service may be temporarily unavailable.
• Order service doesn’t fail immediately.
• Better reliability.
• Retry support.
• Loose coupling.

Interview Tip: Use REST for immediate responses and Kafka for asynchronous business workflows.

Saga Pattern manages distributed transactions across multiple microservices.

Instead of one database transaction,

each service performs:

• Local transaction
• Publish event

If one step fails,

Compensating events are triggered.

Example

Order Created

↓

Payment Success

↓

Inventory Reserved

↓

Shipping Created

If Inventory fails

Inventory Failed

↓

Payment Refund

↓

Order Cancelled

Advantages

• No distributed database transaction
• High scalability
• Fault tolerance

Interview Tip: Saga replaces two-phase commit in microservices.

In Event Choreography,

there is no central coordinator.

Each service listens for events and decides what to do.

Example

OrderCreated

↓

Payment Service

↓

PaymentCompleted

↓

Inventory Service

↓

InventoryReserved

↓

Shipping Service

Each service reacts independently.

Advantages

• Decentralized
• Easy to scale
• Loosely coupled

Disadvantages

• Harder to debug
• Event flow becomes complex

Interview Tip: Kafka is commonly used for Saga Choreography.

Business events evolve over time.

Example

Version 1

{
  "orderId":101,
  "amount":500
}

Version 2

{
  "orderId":101,
  "amount":500,
  "coupon":"NEW10"
}

Older consumers should continue working.

Best Practices

• Never remove required fields abruptly.
• Add optional fields with default values.
• Maintain backward compatibility.

Interview Tip: Avoid breaking existing consumers when evolving event schemas.

Schema Evolution allows event structures to change without breaking producers or consumers.

Common approaches:

• Avro + Schema Registry
• Protobuf
• JSON with version fields (less strict)

Example

Version 1

OrderID
Amount

Version 2

OrderID
Amount
Currency

Older consumers continue processing if compatibility rules are respected.

Interview Tip: Avro with Schema Registry is the preferred enterprise solution.

An Idempotent Consumer processes duplicate messages safely.

Suppose

Order Paid

is delivered twice.

Without idempotency

₹1000 deducted twice

With idempotency

Consumer checks

Payment ID

If already processed

↓

Ignore duplicate.

Implementation

Maintain a table

processed_event

event_id

processed_time

Before processing

↓

Check event ID.

Interview Tip: Producer idempotence prevents duplicate writes to Kafka.

Idempotent Consumers prevent duplicate business operations.

The Outbox Pattern ensures database updates and event publishing remain consistent.

Problem

Save Order

↓

Database Success

↓

Kafka Publish Failed

Database updated

Event missing

Inconsistent system.

Solution

Order Table

↓

Outbox Table

↓

Background Publisher

↓

Kafka

Database transaction writes both:

• Order
• Outbox Event

Then

Outbox Publisher sends event to Kafka.

Advantages

• Prevents lost events
• Reliable event publishing
• Transaction consistency

Interview Tip: Outbox Pattern avoids dual-write problems.

CDC captures database changes automatically and publishes them to Kafka.

Instead of changing application code,

database changes become events.

Flow

Database

↓

CDC

↓

Kafka

↓

Consumers

Use Cases

• Database synchronization
• Audit systems
• Search indexing
• Data warehouse updates
• Analytics

Interview Tip: CDC avoids modifying existing applications to publish events.

Debezium is an open-source CDC platform.

It monitors database transaction logs.

Example

MySQL

↓

Binlog

↓

Debezium

↓

Kafka

↓

Consumers

Whenever a row changes,

Debezium publishes an event.

Example

INSERT Order

↓

Kafka Event

Advantages

• No application changes
• Reliable CDC
• Near real-time replication
• Supports MySQL, PostgreSQL, SQL Server, Oracle, MongoDB

Interview Tip: Debezium reads database transaction logs rather than periodically querying tables.

Architecture

Client
   |
Order Service
   |
OrderCreated Topic
   |
---------------------------------------------------
|          |            |             |
Payment  Inventory   Shipping   Notification
   |
PaymentCompleted Topic
   |
Order Service

Flow

1. Customer places order.

2. Order Service stores the order.

3. Order Service publishes OrderCreated.

4. Payment Service processes payment.

5. Payment Service publishes PaymentCompleted.

6. Inventory reserves stock.

7. Shipping creates shipment.

8. Notification sends email/SMS.

9. Order status is updated.

Interview Tip: Mention Outbox Pattern, Saga Pattern, and Dead Letter Topics for a production-grade design.

Production configuration:

acks=all
enable.idempotence=true
replication.factor=3
min.insync.replicas=2

Additionally:

• Manual offset commits
• Multiple brokers
• Proper retention
• Monitor Under Replicated Partitions (URP)

Interview Tip: No single setting prevents message loss. Reliability comes from producer, broker, and consumer working together.

Use:

• Idempotent Producer
• Idempotent Consumer
• Unique Event ID
• Transactional Producer (where appropriate)

Example

processed_events

event_id

If already processed,

↓

Skip processing.

Interview Tip: Exactly-once delivery still requires idempotent business logic.

Use

• @RetryableTopic
• DefaultErrorHandler
• Exponential Backoff
• Dead Letter Topic after retry exhaustion

Flow

Message

↓

Failure

↓

Retry

↓

Retry

↓

DLT

@RetryableTopic(attempts = "3")
@KafkaListener(topics = "order-topic")
public void consume(OrderEvent event) {
    // Business logic
}

@DltHandler
public void handleDeadLetter(OrderEvent event) {
    // Log, alert, or persist for investigation
}

Failed messages are stored in a DLT for later analysis and replay.

Symptoms:

• Consumer Lag increases.
• Processing latency grows.
• SLA violations.

Solutions:

• Increase partitions.
• Add more consumer instances.
• Increase listener concurrency.
• Optimize business logic.
• Scale horizontally.

Interview Tip: Consumer Lag is the first metric to investigate.

Architecture:

• Multiple brokers
• Many partitions
• Multiple consumer instances
• Producer batching
• Compression (zstd or lz4)
• Asynchronous producers
• Horizontal scaling

Example

Producer

↓

100 Partitions

↓

50 Consumers

↓

Parallel Processing

Interview Tip: Kafka scales horizontally through partitions and consumer groups.

• Use the same business key (e.g., OrderId) for related events.
• Route all events for that key to the same partition.
• Avoid changing partition count unless necessary.

Interview Tip: Ordering is guaranteed only within a partition.

• Increase topic partitions.
• Add consumer instances within the same consumer group.
• Configure listener concurrency.
• Deploy multiple application instances.

Interview Tip: Maximum parallelism equals the number of partitions.

A poison message always fails processing.

Solution:

• Retry a limited number of times.
• Send to DLT.
• Investigate and replay after fixing the issue.

Never retry indefinitely.

Interview Tip: Poison messages should never block the main consumer.

Use a unique event ID.

Processing Flow

Receive Event

↓

Check processed_event table

↓

Already Exists?

↓

Yes → Ignore

↓

No → Process → Save Event ID

This prevents duplicate business operations.

Use:

• Avro + Schema Registry (preferred)
• Optional fields with defaults
• Backward-compatible schema changes
• Event version field if needed

Never remove mandatory fields without a migration strategy.

Interview Tip: Schema evolution should not break existing consumers.

Migration strategy:

1. Keep REST and Kafka running in parallel.

2. Publish events after successful REST operations.

3. Update downstream services to consume Kafka events.

4. Gradually reduce REST dependencies.

5. Remove synchronous calls after validation.

Interview Tip: Incremental migration is safer than a big-bang migration.

• Use Schema Registry.
• Enforce backward compatibility.
• Add optional fields with defaults.
• Deploy producers and consumers independently.

Interview Tip: Avoid incompatible schema changes in production.

Use:

• Idempotent Producer
• Transactional Producer
• isolation.level=read_committed
• Idempotent Consumer using Payment ID or Transaction ID
• Manual offset commit after successful business processing

Interview Tip: Exactly-once requires Kafka features and idempotent business logic.

Steps:

1. Check Consumer Lag metrics.

2. Verify consumer health.

3. Review application logs.

4. Check database/API latency.

5. Inspect broker performance.

6. Verify partition distribution.

7. Scale consumers if required.

Interview Tip: Consumer Lag is usually a symptom. Find the underlying bottleneck.

Kafka automatically:

1. Detects broker failure.

2. Elects a new leader from the ISR.

3. Updates producer and consumer metadata.

4. Continues processing.

After recovery:

• Verify ISR.
• Check Under Replicated Partitions.
• Ensure replication catches up.

Interview Tip: High availability depends on replication and ISR-based leader election.

Options:

• Reset consumer group offsets.
• Create a new consumer group.
• Use seek() to move to an earlier offset.

Example

consumer.seek(topicPartition, 0);

Use Cases

• Bug fixes
• Data recovery
• Analytics rebuild
• Cache reconstruction

A single listener can subscribe to multiple topics.

@KafkaListener(topics = {
    "order-topic",
    "payment-topic",
    "inventory-topic"
})
public void consume(String event) {
    // Process based on topic
}

Alternatively, use dedicated listeners for different domains to keep business logic clean.

Interview Tip: Separate listeners are often easier to maintain for complex business domains.

Use a Correlation ID (or Trace ID).

Flow

Order Service

↓

Correlation ID: abc123

↓

Kafka

↓

Payment Service

↓

Inventory Service

↓

Shipping Service

↓

Notification Service

Every log entry contains the same Correlation ID.

Also integrate:

• OpenTelemetry
• Micrometer
• Prometheus
• Grafana
• Jaeger or Zipkin (for distributed tracing)

Interview Tip: Always include a Correlation ID in Kafka message headers to trace requests across microservices.

Producer Service

@Service
public class OrderProducer {

    private final KafkaTemplateu003cString, OrderEventu003e kafkaTemplate;

    public OrderProducer(KafkaTemplateu003cString, OrderEventu003e kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public void publish(OrderEvent event) {
        kafkaTemplate.send("order-topic", event.getOrderId().toString(), event);
    }
}

REST Controller

@RestController
@RequestMapping("/orders")
public class OrderController {

    private final OrderProducer producer;

    public OrderController(OrderProducer producer) {
        this.producer = producer;
    }

    @PostMapping
    public String createOrder(@RequestBody OrderEvent order) {

        producer.publish(order);

        return "Order Published";
    }
}

Interview Tip: Always use a business key (Order ID, Customer ID) to maintain ordering.

@Service
public class OrderConsumer {

    @KafkaListener(
            topics = "order-topic",
            groupId = "order-group")
    public void consume(OrderEvent order){

        System.out.println(order);

    }
}

Interview Tip: @KafkaListener automatically handles polling, deserialization, threading, and offset management.

DTO

public class OrderEvent {

    private Long orderId;
    private Double amount;

    // getters and setters

}

Producer Configuration

spring.kafka.producer.value-serializer=org.springframework.kafka.support.serializer.JsonSerializer

Publish

kafkaTemplate.send("order-topic", orderEvent);

Interview Tip: Spring automatically converts the POJO to JSON using JsonSerializer.

Consumer Configuration

spring.kafka.consumer.value-deserializer=org.springframework.kafka.support.serializer.JsonDeserializer
spring.kafka.consumer.properties.spring.json.trusted.packages=*

Consumer

@KafkaListener(topics = "order-topic")
public void consume(OrderEvent event){

    System.out.println(event.getOrderId());

}

Interview Tip: Producer and consumer must agree on the event schema.

Configuration

spring.kafka.listener.ack-mode=manual
spring.kafka.consumer.enable-auto-commit=false

Consumer

@KafkaListener(topics = "order-topic")
public void consume(OrderEvent order,
                    Acknowledgment acknowledgment){

    processOrder(order);

    acknowledgment.acknowledge();

}

Why?

Offsets are committed only after successful business processing.

Interview Tip: Manual commits reduce the risk of losing messages during failures.

@RetryableTopic(attempts = "3")
@KafkaListener(topics = "order-topic")
public void consume(OrderEvent order){

    throw new RuntimeException("Processing Failed");

}

@DltHandler
public void processDeadLetter(OrderEvent order){

    System.out.println("DLT : " + order);

}

Flow

Topic

↓

Retry

↓

Retry

↓

Retry

↓

Dead Letter Topic

Interview Tip: DLTs isolate permanently failing messages without blocking the main topic.

@RetryableTopic(
    attempts = "5",
    backoff = @Backoff(
            delay = 1000,
            multiplier = 2.0
    )
)
@KafkaListener(topics = "order-topic")
public void consume(OrderEvent order){

}

Retry Pattern

1 sec

↓

2 sec

↓

4 sec

↓

8 sec

Interview Tip: Exponential backoff prevents overwhelming downstream systems during outages.

CompletableFutureu003cSendResultu003cString, OrderEventu003eu003e future =
        kafkaTemplate.send(
                "order-topic",
                orderEvent);

future.whenComplete((result, ex) -u003e {

    if(ex == null){

        System.out.println("Success");

    }else{

        System.out.println("Failure");

    }

});

Benefits

• Non-blocking
• Higher throughput
• Better scalability

Interview Tip: KafkaTemplate.send() is asynchronous in modern Spring Kafka and returns a CompletableFuture.

Custom Partitioner

public class OrderPartitioner implements Partitioner {

    @Override
    public int partition(
            String topic,
            Object key,
            byte[] keyBytes,
            Object value,
            byte[] valueBytes,
            Cluster cluster) {

        String orderId = (String) key;

        return Integer.parseInt(orderId) % 3;
    }

    @Override
    public void close() {

    }

    @Override
    public void configure(Mapu003cString, ?u003e configs) {

    }
}

Configuration

spring.kafka.producer.properties.partitioner.class=com.example.OrderPartitioner

Interview Tip: Use custom partitioners only when the default key-based partitioning does not satisfy business requirements.

enable.idempotence=true
acks=all
retries=2147483647
max.in.flight.requests.per.connection=5

Benefits

• No duplicate writes due to retries.
• Safer producer retries.
• Better reliability.

Interview Tip: enable.idempotence=true prevents duplicate records caused by producer retries but does not eliminate duplicate business processing.

@Bean
public DefaultErrorHandler errorHandler() {

    FixedBackOff backOff =
            new FixedBackOff(3000, 3);

    return new DefaultErrorHandler(backOff);

}

Flow

Message

↓

Exception

↓

Retry

↓

Retry

↓

Dead Letter Topic

(with a DeadLetterPublishingRecoverer configured if DLT publishing is desired)

Interview Tip: DefaultErrorHandler centralizes retry, recovery, and backoff handling.

@Service
public class InventoryConsumer {

    @KafkaListener(
            topics = "order-topic",
            groupId = "inventory-group")
    public void inventory(OrderEvent order){

    }

}

@Service
public class NotificationConsumer {

    @KafkaListener(
            topics = "order-topic",
            groupId = "notification-group")
    public void notification(OrderEvent order){

    }

}

Flow

Order Topic

↓

Inventory Consumer

↓

Notification Consumer

↓

Analytics Consumer

Each consumer group receives its own copy of the message.

Interview Tip: Different consumer groups maintain independent offsets.

Maven Dependency

u003cdependencyu003e
    u003cgroupIdu003eorg.springframework.kafkau003c/groupIdu003e
    u003cartifactIdu003espring-kafka-testu003c/artifactIdu003e
    u003cscopeu003etestu003c/scopeu003e
u003c/dependencyu003e

Test

@SpringBootTest
@EmbeddedKafka(partitions = 1, topics = "order-topic")
class KafkaIntegrationTest {

    @Autowired
    private KafkaTemplateu003cString, Stringu003e kafkaTemplate;

    @Test
    void testKafka() {

        kafkaTemplate.send("order-topic", "Hello Kafka");

    }

}

Benefits

• No external Kafka cluster required.
• Fast automated testing.
• Suitable for CI/CD pipelines.

Interview Tip: @EmbeddedKafka is commonly used for integration testing of Spring Kafka applications.

Although Kafka is primarily asynchronous, Spring Kafka provides request-reply support.

Configuration

Create a ReplyingKafkaTemplate.

Request Producer

ProducerRecordu003cString, OrderRequestu003e request =
        new ProducerRecordu003cu003e("order-request", requestDto);

RequestReplyFutureu003cString, OrderRequest, OrderResponseu003e future =
        replyingKafkaTemplate.sendAndReceive(request);

OrderResponse response = future.get().value();

Consumer

@KafkaListener(topics = "order-request")
@SendTo("order-response")
public OrderResponse process(OrderRequest request) {

    return new OrderResponse("SUCCESS");

}

Flow

Client

↓

Request Topic

↓

Consumer

↓

Response Topic

↓

Client

Use Cases

• Quote generation
• Validation services
• Inventory availability
• Payment authorization

Interview Tip: Kafka is optimized for asynchronous messaging. Use the request-reply pattern only when a response is genuinely required; otherwise, prefer event-driven communication.