Area 07 · Deployment · Concept 13 of 20

Load Balancing

Spread requests across instances so no single one gets overloaded.

11–20 · Production levelPhase 5 · Weeks 18–20≈ 45 min

The idea

With several instances, something must decide which one handles each request. A load balancer spreads traffic using strategies such as round robin, least connections or latency-based routing, and stops sending traffic to instances that fail health checks.

LLM traffic is uneven: one request may stream for 30 seconds while another finishes in 200 ms. Least-connections or least-outstanding-requests usually beats round robin. For streaming, make sure idle timeouts are long enough and buffering is off.

For self-hosted model servers, smarter balancers route by prefix (to reuse KV cache) or by queue depth.

Why it matters

Bad balancing makes some users wait while other instances sit idle.

Words to know

  • Round robinSend requests to instances in turn.
  • Least connectionsSend to the instance with the fewest active requests.

Try it without code

Simulate

  1. Draw 3 servers.
  2. Assign 10 requests with random durations using round robin, then least connections.
  3. Compare the busiest server’s load.

Use it in code

nginx in front of three API instances, tuned for streaming.

nginx
upstream rag_api {
    least_conn;
    server api1:8000 max_fails=3 fail_timeout=10s;
    server api2:8000 max_fails=3 fail_timeout=10s;
    server api3:8000 max_fails=3 fail_timeout=10s;
}
server {
    listen 80;
    location / {
        proxy_pass http://rag_api;
        proxy_http_version 1.1;
        proxy_buffering off;          # needed for SSE streaming
        proxy_read_timeout 300s;      # long LLM responses
    }
}
PRACTICE · TICK IT AFTER THIS

Run three API replicas behind nginx with Docker Compose and confirm streaming still works.

Common mistakes

  • Proxy buffering breaking SSE.
  • 60-second idle timeouts cutting long generations.

Go deeper