Kafka Quick Recap

Producer
   ↓
Topic
   ↓
Partitions
   ↓
Consumer Group
   ↓
Consumers

1. Producer

• Publishes messages/records to Kafka.
• Can publish to one or multiple topics.
• Producer doesn’t own partitions.
• Kafka decides which partition receives the record.

Producer → Topic A
         → Topic B
         → Topic C

2. Topic

• Logical category/name for messages.
• A topic contains one or more partitions.

order-events
 ├── P0
 ├── P1
 └── P2

3. Partition

• Actual place where Kafka stores records.
• Records have offsets within a partition.
• Ordering is guaranteed within a partition, not across the whole topic.

P0 → Offset 0, 1, 2, 3, 4...

4. Offset

• Position of a record inside a partition.
• Offset starts from 0.
• Current Offset usually represents the next position to read.

Records:  0  1  2  3  4  5
Next:                         6

So if:

Current Offset = 6
Log-End Offset = 6
Lag = 0

the consumer is caught up.

5. Consumer

• Reads records from Kafka.
• Can consume from multiple topics.
• Kafka assigns partitions to consumers.

Consumer → P0
         → P1

6. Consumer Group

• A team of consumers working together.
• A consumer group can consume from multiple topics.
• Within the same consumer group:

One partition → only one consumer at a time.

Example:

inventory-group

Consumer 1 → P0, P1
Consumer 2 → P2

7. Multiple Consumer Groups

Different groups can consume the same topic/partition independently.

                 order-events
                /      |      \
               P0     P1      P2
                │       │       │
       ┌────────┴───────┴───────┴────────┐
       │                                  │
inventory-group                    analytics-group
       │                                  │
   Consumers                          Consumers

8. Scaling Rule ⭐

3 partitions + 1 consumer
→ consumer handles all 3

3 partitions + 3 consumers
→ each can handle 1

3 partitions + 5 consumers
→ 2 consumers will be idle

Maximum useful parallelism within one consumer group = number of partitions.

The 7 things to remember

Concept	Remember
Producer	Publishes records
Topic	Category/logical stream
Partition	Stores ordered records
Offset	Record position in partition
Consumer	Reads records
Consumer Group	Consumers working together
Lag	Records still waiting to be consumed

⭐ One-line mental model

Producer publishes → Topic contains partitions → Partitions contain records with offsets → Consumer Groups consume those partitions → Consumers are assigned partitions.