Kafka earliest vs latest — Quick Recap

1. Consumer Group

A consumer group is identified by a group-id.

spring:
  kafka:
    consumer:
      group-id: inventory-group

You don’t declare:

number of groups = 1

Instead, you give consumers a group ID.

inventory-group       → Inventory Service
notification-group    → Notification Service

Different group-ids = different consumer groups.

2. First time Inventory Service starts

Suppose:

Kafka
Order1
Order2
Order3
Order4

Inventory starts with:

group-id: inventory-group

If inventory-group has never consumed before, Kafka has no committed offset for this group.

Now auto-offset-reset decides where to start.

3. earliest

auto-offset-reset: earliest

Means:

“If I don’t have a previous offset, start from the earliest message still available.”

Example:

Kafka:

0 → Order1
1 → Order2
2 → Order3
3 → Order4

Inventory starts
      ↓
earliest
      ↓
Order1 → Order2 → Order3 → Order4

So it can process old + new messages.

4. latest

auto-offset-reset: latest

Means:

“If I don’t have a previous offset, start at the current end and wait for new messages.”

Example:

Kafka:

0 → Order1
1 → Order2
2 → Order3
3 → Order4
              ↑
             END

Inventory starts with latest
              ↓
        wait for new events

If Order5 arrives:

Order5 → Inventory ✅

The existing Order1–4 are not consumed by this new consumer group.

5. Very important: Restarting the service

Suppose Inventory has already consumed:

Order1
Order2
Order3

and Kafka has a committed offset for:

inventory-group

Now Inventory crashes.

While it’s down:

Order4
Order5
Order6

arrive.

When Inventory starts again:

Kafka
 ↓
Find inventory-group
 ↓
Find committed offset
 ↓
Resume from that position

It normally processes:

Order4 → Order5 → Order6

It does NOT use latest to skip them.

So:

latest does NOT mean “skip messages whenever my service is down.”

It matters when there is no valid committed offset.

6. Offset

An offset is the position of a record inside a Kafka partition.

Example:

Partition 0

Offset 0 → Order1
Offset 1 → Order2
Offset 2 → Order3
Offset 3 → Order4

You don’t declare the offset count yourself.

Kafka assigns offsets automatically.

7. Offset increases

For a particular partition, offsets increase as new records are appended:

0 → 1 → 2 → 3 → 4 → 5 → 6 → ...

For a long-running application, offsets can become:

millions
billions
even much larger

That’s completely normal.

8. Offset is per partition

If your topic has two partitions:

Partition 0          Partition 1

0 → Order1           0 → Order2
1 → Order3           1 → Order4
2 → Order5           2 → Order6

Both partitions can have offset 0.

So:

Offset is NOT a global message number.

It is the position within that partition.

9. Retention doesn’t reset offsets

Suppose Kafka had:

0  1  2  3  4  5  6

Old messages are deleted because of retention:

2  3  4  5  6

Kafka does not renumber them:

❌ 0 1 2 3 4

They remain:

✅ 2 3 4 5 6

So offsets represent positions in the log and can keep increasing.

⭐ Most Important Mental Model

Remember this flow:

              Consumer starts
                    │
                    ▼
       Does consumer group already
        have a valid committed offset?
              /              \
            YES              NO
             │                │
             ▼                ▼
       Use committed      auto-offset-reset
          offset               │
                         ┌─────┴─────┐
                         ▼           ▼
                     earliest      latest
                         │           │
                         ▼           ▼
                    Oldest       Current end
                    available    / wait for
                    message       new events

One-line memory trick

earliest → 🕐 “Give me the old available messages.”

latest → 🆕 “Start from now and give me new messages.”

Committed offset exists → 📍 “Continue from where my group stopped.”

Offset → 🔢 “Position of a record inside a partition.”