Retry Storms: The Silent Enemy of Scalable Microservices

Retry Storm is a situation where multiple clients/services keep retrying failed requests aggressively, creating even more load on an already struggling service, which can eventually bring down the entire system.

Simple Example

Suppose:

• Service A calls Service B
• Service B is overloaded and starts responding slowly
• Service A has retry logic configured for 3 retries

Now imagine:

• 1,000 requests arrive at Service A
• Service B fails for all requests
• Service A retries each request 3 times

Instead of 1,000 requests, Service B now receives:

Original Requests = 1,000
Retries = 3,000

Total = 4,000 requests

The overloaded service gets 4x more traffic, making recovery even harder.

---

Microservices Example

User Request
    |
API Gateway
    |
Order Service
    |
Payment Service
    |
Bank Service

If Bank Service is down:

• Payment Service retries 3 times
• Order Service retries 3 times
• API Gateway retries 3 times

For a single request:

Gateway Retry = 3
Order Retry = 3
Payment Retry = 3

Total attempts = 3 × 3 × 3 = 27

One user request may generate 27 requests to the failing service.

For 1,000 users:

1000 × 27 = 27,000 requests

This is a classic Retry Storm.

---

Problems Caused

✅ Increased CPU usage

✅ Increased memory consumption

✅ Thread pool exhaustion

✅ Database connection exhaustion

✅ Network congestion

✅ Cascading failures

✅ Complete system outage

---

How to Prevent Retry Storms

1. Exponential Backoff

Instead of:

Retry after 1s
Retry after 1s
Retry after 1s

Use:

Retry after 1s
Retry after 2s
Retry after 4s
Retry after 8s

This gives the failing service time to recover.

---

2. Add Jitter

Without jitter:

1000 services retry exactly at 2 seconds

With jitter:

Service 1 -> 2.1 sec
Service 2 -> 2.7 sec
Service 3 -> 3.0 sec

Requests are spread out.

---

3. Circuit Breaker

Using Resilience4j:

Failure Rate > Threshold
       ↓
Circuit Opens
       ↓
No More Requests Sent

This stops retries completely for a period.

---

4. Retry Only Transient Errors

Retry:

503 Service Unavailable
504 Gateway Timeout
Network Timeout

Do NOT retry:

400 Bad Request
401 Unauthorized
404 Not Found
Validation Errors

---

5. Limit Maximum Retries

Bad:

10 retries

Good:

2-3 retries

---

Interview Answer (Short)

A Retry Storm occurs when multiple services or clients continuously retry failed requests, causing a huge increase in traffic on an already overloaded service. This can lead to cascading failures, thread exhaustion, and system outages. It is prevented using exponential backoff, jitter, circuit breakers, and limited retries.