Kafka Keeps Messages After Consumption? Here’s Why Big Tech Needs It

Absolutely. This is actually the main reason Kafka became so popular.

Many developers initially think:

“Why store old events? Once Inventory Service processes the order, why keep it?”

In small systems, you don’t need it.

In large-scale systems, retaining events becomes extremely valuable.

---

Use Case 1: New Service Added Later

Suppose your ecommerce platform has:

Order Service
    |
    +--> Inventory Service
    +--> Notification Service

After 1 year, the business says:

“We need an Analytics Service to analyze customer buying patterns.”

With RabbitMQ:

• Old messages are gone.
• Analytics starts from today.

With Kafka:

• Last year’s order events still exist (depending on retention).
• Analytics can replay all historical events.

Order Events
    |
    +--> Analytics Service (added later)

No changes needed in Order Service.

---

Use Case 2: Service Failure Recovery

Imagine:

Order Created
      |
      v
Inventory Service

Inventory Service is down for 2 hours.

With Kafka:

• Events remain in Kafka.
• When Inventory Service comes back, it continues from the last processed offset.

Order #100
Order #101
Order #102
Order #103

Nothing is lost.

---

Use Case 3: Audit Trail

Banking systems often need:

Account Created
Money Deposited
Money Withdrawn
Loan Approved

Years later, auditors ask:

“Show every transaction related to this account.”

Kafka can retain all events for long periods.

This creates a complete history.

---

Use Case 4: Event Sourcing

This is a common system design interview topic.

Instead of storing only current state:

Current Balance = ₹10,000

Store every event:

Account Created
+5000
+3000
-1000
+3000

Current balance is reconstructed from events.

This pattern is called:

Event Sourcing

Kafka is frequently used here.

---

Use Case 5: Machine Learning & Recommendations

Suppose you want:

Amazon-like recommendations

Store events:

User Viewed Product
User Added To Cart
User Purchased Product

Later:

Recommendation Engine

reads historical events and generates recommendations.

---

Use Case 6: Real-Time Analytics

Companies like:

• Netflix
• Uber
• LinkedIn

generate millions of events:

Video Started
Video Paused
Ride Requested
Ride Completed
Profile Viewed

Kafka streams these events into analytics systems.

---

In Your Local Store Ecommerce Project

Suppose you build:

User Service
Product Service
Order Service
Inventory Service
Notification Service

Order Service publishes:

{
  "orderId": 1001,
  "userId": 50,
  "amount": 1200
}

Kafka stores this event.

Today:

• Inventory Service consumes it.
• Notification Service consumes it.

6 months later:

• Fraud Detection Service added.
• Analytics Service added.
• Recommendation Service added.

All can consume historical order events.

No changes to Order Service.

---

Do Most Companies Keep Events Forever?

Not usually.

Common retention periods:

• 7 days
• 30 days
• 90 days

Critical systems may archive events to:

• Amazon Web Services S3
• Data warehouses
• Data lakes

Kafka acts as the event backbone.

---

Interview Answer

If an interviewer asks:

“Why does Kafka retain events?”

You can answer:

Kafka retains events so that multiple consumers can process the same event independently, failed consumers can replay missed events, new services can consume historical data, and organizations can support analytics, auditing, event sourcing, and machine learning use cases without modifying existing producers.