The idea
Kubernetes runs containers across a cluster of machines and keeps them in the state you declare. A Pod is one or more containers scheduled together. A Deployment says "keep 3 replicas of this pod running with this image" and handles rolling updates. A Service gives pods a stable network name and load-balances across them.
You describe resources in YAML and apply them with kubectl. Kubernetes restarts crashed containers, replaces failed nodes, scales replicas and rolls out new versions gradually.
It is powerful and complex. Many AI teams start on simpler platforms (Cloud Run, ECS, Fly) and move to Kubernetes for GPU scheduling, many services or strict control.
Most large companies run on Kubernetes, and GPU model serving at scale usually means Kubernetes.
Picture it
Words to know
- PodSmallest deployable unit: one or more containers.
- DeploymentManages replicas and rollouts of pods.
- ServiceStable address and load balancing for pods.
Try it without code
Interactive tutorial
- Open the Kubernetes Basics tutorial.
- Create a deployment, expose it, scale it and update it in the browser sandbox.
Use it in code
Deployment and Service for the RAG API.
apiVersion: apps/v1
kind: Deployment
metadata: { name: rag-api }
spec:
replicas: 2
selector: { matchLabels: { app: rag-api } }
template:
metadata: { labels: { app: rag-api } }
spec:
containers:
- name: api
image: ghcr.io/you/rag-api:1.4.0
ports: [{ containerPort: 8000 }]
envFrom: [{ secretRef: { name: rag-api-secrets } }]
resources: { requests: { cpu: 250m, memory: 512Mi }, limits: { memory: 1Gi } }
readinessProbe: { httpGet: { path: /ready, port: 8000 } }
livenessProbe: { httpGet: { path: /health, port: 8000 } }
---
apiVersion: v1
kind: Service
metadata: { name: rag-api }
spec:
selector: { app: rag-api }
ports: [{ port: 80, targetPort: 8000 }]Run your API on a local kind or minikube cluster with 2 replicas, then kill a pod and watch it come back.
Common mistakes
- No resource requests, so scheduling is unpredictable.
- Using :latest tags, so you cannot tell what is running.