How load balancer implemented in enterprise application?

In enterprise applications, a load balancer is almost never part of your application code. It is an infrastructure component that sits in front of your application instances and distributes incoming requests.

Let’s understand it from a real-world Java Spring Boot microservices perspective.

---

Scenario

Suppose you have an e-commerce application.

Internet
    |
Customers

One Spring Boot application is running.

                Spring Boot
                  Product Service

This works for 100 users.

But tomorrow 50,000 users visit.

One server cannot handle all requests.

So you create multiple instances.

Product Service

Instance 1
Instance 2
Instance 3
Instance 4

Now the question is…

How does the client know which instance to call?

The answer is:

Load Balancer

---

Without Load Balancer

Client
   |
   |------> Product Service Instance 1

If Instance 1 crashes…

Client
   |
   |------> Dead Server

Application fails.

---

With Load Balancer

                 Client
                    |
                    |
             Load Balancer
          /      |      \
         /       |       \
     Instance1 Instance2 Instance3

The client never knows the backend servers.

It only knows

https://api.company.com

The load balancer decides where to send the request.

---

How enterprise applications implement it

There are multiple layers.

---

1. External Load Balancer

This receives traffic from the internet.

Examples

• NGINX
• HAProxy
• Traefik
• AWS Application Load Balancer
• Azure Application Gateway

Example

Internet
      |
      |
AWS ALB
      |
      |
Kubernetes Cluster

---

2. Kubernetes Service

Suppose your Product Service has 5 Pods.

Pod 1

Pod 2

Pod 3

Pod 4

Pod 5

Kubernetes creates a Service.

Product Service

ClusterIP
10.96.0.15

All requests go to

10.96.0.15

Internally Kubernetes forwards requests.

          Service

        10.96.0.15
       /   |   |   \
      /    |   |    \
Pod1 Pod2 Pod3 Pod4

This is internal load balancing.

You don’t write any code.

---

How Kubernetes knows available pods?

Every pod registers automatically.

Deployment

↓

Creates Pods

↓

Service watches labels

↓

Service updates endpoints

If Pod 3 dies

Pod1

Pod2

Pod4

Pod5

Service removes Pod3 automatically.

No configuration change.

---

Where API Gateway fits?

Internet

    |

Load Balancer

    |

API Gateway

    |

--------------------------
|      |        |        |
User  Cart   Product  Order

The API Gateway itself is also deployed in multiple pods.

Example

              Load Balancer

                  |

      ------------------------

      |          |          |

 Gateway1   Gateway2   Gateway3

The load balancer distributes requests among gateway instances.

Then the gateway routes to services.

---

How Product Service calls Order Service?

Suppose Product Service wants Order Service.

It does not call

192.168.0.10

or

localhost

Instead it calls

http://order-service

Kubernetes DNS resolves it.

order-service

↓

ClusterIP

↓

Load Balancing

↓

Order Pod1

Order Pod2

Order Pod3

Every request may reach a different pod.

---

What if a pod crashes?

Suppose

Order

Pod1

Pod2

Pod3

Pod2 crashes.

Order

Pod1

Pod3

The Service removes it automatically.

Future requests go only to healthy pods.

---

How health is checked?

Kubernetes continuously checks

• Liveness Probe
• Readiness Probe

Only pods passing the readiness probe receive traffic.

        Service

      /    |     \

Healthy Healthy Dead

  ✓        ✓      ✗

Dead pod gets zero traffic.

---

Which algorithm decides where to send requests?

Common algorithms include:

Round Robin (most common)

Request1 -> Pod1

Request2 -> Pod2

Request3 -> Pod3

Request4 -> Pod1

Request5 -> Pod2

---

Least Connections

Pod1 : 120 connections

Pod2 : 30 connections

↓

Next request goes to Pod2

---

Random

Request

↓

Random Pod

---

IP Hash

A user’s IP address is hashed so the same client is consistently routed to the same backend (useful for session affinity).

---

Do Spring Boot applications implement load balancing?

Usually no for incoming traffic.

Spring Boot only exposes REST APIs.

Infrastructure performs the balancing.

Earlier, Spring Cloud used client-side load balancing with components like Ribbon (now retired) and Spring Cloud LoadBalancer for service-to-service calls. In modern Kubernetes environments, most incoming load balancing is handled by Kubernetes Services and cloud load balancers, while outgoing service discovery can use Kubernetes DNS, optionally with Spring Cloud LoadBalancer.

---

End-to-end enterprise architecture

                 Internet
                     |
             Cloud Load Balancer
                     |
              Ingress Controller
                     |
          ------------------------
          |          |           |
      API Gateway API Gateway API Gateway
          |          |           |
          ------------------------
                     |
        ---------------------------------
        |         |         |           |
     User Svc  Cart Svc  Order Svc  Product Svc
        |         |         |           |
      Service   Service   Service    Service
        |         |         |           |
      Pods      Pods      Pods        Pods
        |         |         |           |
      Database  Redis   RabbitMQ    Kafka

Key takeaways

• Cloud Load Balancer (or NGINX/HAProxy) distributes external traffic to API Gateway or Ingress.
• API Gateway handles authentication, routing, rate limiting, and other cross-cutting concerns.
• Kubernetes Service load-balances requests across healthy pods of a microservice.
• Kubernetes DNS lets services communicate using names like http://order-service instead of IP addresses.
• Readiness probes ensure only healthy pods receive traffic.
• Developers typically do not write load-balancing logic in application code; it is provided by the infrastructure.

This is the architecture you’ll commonly see in enterprise deployments on platforms such as Kubernetes running in cloud environments.