Kafka Topic Retention Explained with Real-World Examples

This is actually one of Kafka’s biggest strengths.

The concept you’re referring to is Kafka Topic Retention.

Instead of deleting a message immediately after it is consumed (like RabbitMQ typically does), Kafka stores events for a configurable retention period, even after consumers have read them.

For example:

• 7 days
• 30 days
• 1 year
• Even forever (until disk is full or manually deleted)

---

Example

Suppose an e-commerce application has an orders topic.

Order Service
      |
      v
+----------------------+
| orders Topic         |
|----------------------|
| Order #101           |
| Order #102           |
| Order #103           |
| Order #104           |
+----------------------+

Today,

• Inventory Service consumes Order #101
• Notification Service consumes Order #101
• Billing Service consumes Order #101

Normally you’d think Kafka deletes it.

It doesn’t.

It remains inside the topic until the retention period expires.

---

Why keep events?

1. Analytics

Months later the business team asks:

“How many orders were placed during Diwali last year?”

No need to query operational databases.

Simply read historical events.

Orders Topic (1 year retention)

2025 Orders
2026 Orders
2027 Orders

Analytics Service
        |
        v
Calculate sales
Top products
Revenue
Customer trends

---

2. Replay Events

Imagine Inventory Service had a bug.

Yesterday it processed orders incorrectly.

You fix the bug.

Instead of asking customers to place orders again, simply replay the events.

Kafka

Order1
Order2
Order3
Order4

↓

Reset consumer offset

↓

Read again

↓

Inventory updated correctly

This is called event replay.

---

3. New Consumer

Suppose six months later your company develops a Recommendation Service.

You don’t need Order Service to resend every order.

Simply create a new consumer.

Orders Topic

Order1
Order2
Order3
Order4

               New Recommendation Service
                         |
                         v
               Reads all historical orders

Very powerful.

---

4. Machine Learning

Data scientists may want

• buying patterns
• fraud detection
• recommendations

They can consume historical events directly.

---

How long can Kafka keep events?

Kafka retention is configurable.

Example:

retention.ms = 604800000

= 7 days

or

retention.ms = 31536000000

= 1 year

or

retention.ms = -1

= Never delete based on time.

---

Does Kafka keep data for years?

Yes.

Many companies configure Kafka to retain data for months or years.

However…

Kafka is not usually intended as your permanent data warehouse.

Keeping years of data requires significant storage and increases operational costs.

A common architecture is:

                Kafka
                  |
                  |
        --------------------
        |        |         |
 Inventory   Notification  Payment

                  |
                  v
          Data Lake / Warehouse
      (S3, HDFS, BigQuery, Snowflake)

                  |
                  v
          Analytics / BI / ML

In this setup:

• Kafka retains events for days or weeks (sometimes months).
• A sink connector (often using Kafka Connect) continuously copies events to long-term storage like object storage or a data warehouse.
• Analytics teams query the data lake or warehouse rather than Kafka itself.

---

Interview Answer (30 seconds)

Kafka stores events in topics even after consumers have read them. This is controlled using topic retention policies. Events can be retained for days, months, or even years depending on business requirements. Long retention enables event replay, onboarding new consumers, analytics, auditing, and machine learning use cases. In production, Kafka is often used as an event streaming platform, while older events are archived to data lakes or data warehouses for long-term storage.

This is an excellent interview topic because it demonstrates that you understand Kafka is an event log, not just a message queue.