The idea
With several instances, something must decide which one handles each request. A load balancer spreads traffic using strategies such as round robin, least connections or latency-based routing, and stops sending traffic to instances that fail health checks.
LLM traffic is uneven: one request may stream for 30 seconds while another finishes in 200 ms. Least-connections or least-outstanding-requests usually beats round robin. For streaming, make sure idle timeouts are long enough and buffering is off.
For self-hosted model servers, smarter balancers route by prefix (to reuse KV cache) or by queue depth.
Bad balancing makes some users wait while other instances sit idle.
Words to know
- Round robinSend requests to instances in turn.
- Least connectionsSend to the instance with the fewest active requests.
Try it without code
Simulate
- Draw 3 servers.
- Assign 10 requests with random durations using round robin, then least connections.
- Compare the busiest server’s load.
Use it in code
nginx in front of three API instances, tuned for streaming.
upstream rag_api {
least_conn;
server api1:8000 max_fails=3 fail_timeout=10s;
server api2:8000 max_fails=3 fail_timeout=10s;
server api3:8000 max_fails=3 fail_timeout=10s;
}
server {
listen 80;
location / {
proxy_pass http://rag_api;
proxy_http_version 1.1;
proxy_buffering off; # needed for SSE streaming
proxy_read_timeout 300s; # long LLM responses
}
}Run three API replicas behind nginx with Docker Compose and confirm streaming still works.
Common mistakes
- Proxy buffering breaking SSE.
- 60-second idle timeouts cutting long generations.