Area 07 · Deployment · Concept 12 of 20

Autoscaling

Add or remove instances automatically as traffic goes up and down.

11–20 · Production levelPhase 5 · Weeks 18–20≈ 1 hour

The idea

Traffic changes through the day. Autoscaling adds instances when load rises and removes them when it falls, so you pay for what you use and stay fast at peaks. Platforms scale on CPU, memory, concurrent requests, queue length or custom metrics.

For AI services, CPU is often a poor signal: your API mostly waits on model calls. Scale on concurrent requests or queue depth instead. For GPU servers, scale on queue length or KV cache utilisation, and remember new GPU instances take minutes to start and load weights.

Set sensible minimums (avoid cold starts on critical paths) and maximums (protect your budget and provider rate limits).

Why it matters

It keeps latency steady under load without paying for peak capacity all day.

Words to know

  • HPAHorizontal Pod Autoscaler in Kubernetes.
  • Scale-out / scale-inAdding / removing instances.

Try it without code

Pick a signal

  1. For your API and your worker, write the metric that best shows they are overloaded.
  2. Write min and max instance counts for each.

Use it in code

Kubernetes HPA scaling the API on CPU, and workers on queue length (with KEDA).

yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: rag-api }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: rag-api }
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 60 } }
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: ingest-worker }
spec:
  scaleTargetRef: { name: ingest-worker }
  minReplicaCount: 0
  maxReplicaCount: 8
  triggers:
    - type: redis
      metadata: { address: redis:6379, listName: rq:queue:ingest, listLength: "5" }
PRACTICE · TICK IT AFTER THIS

Configure autoscaling for your API on your platform and run a load test that triggers a scale-out.

Common mistakes

  • Scaling on CPU for I/O-bound services.
  • No max, so a traffic spike becomes a cost spike.

Go deeper