The idea
Serverless platforms run your code when a request or event arrives and scale to zero when idle. You pay per invocation and duration, with no servers to patch. Serverless containers (Cloud Run, Azure Container Apps) give the same model for whole Docker images.
They fit AI apps well for API endpoints that call hosted models, webhooks and light background jobs. Limits matter: maximum execution time, memory, cold-start latency, and streaming support vary by platform. Long agent runs and GPU work usually need other options.
Keep functions stateless, keep state in databases and queues, and watch cold starts on latency-sensitive paths.
It is often the cheapest and fastest way to put an AI API online for a portfolio project or an early product.
Words to know
- Cold startExtra latency when a new instance starts from zero.
- Scale to zeroNo running instances (and no cost) when idle.
Try it without code
Deploy a function
- Create a free account on a serverless platform.
- Deploy a hello-world function.
- Call it twice and compare cold vs warm latency.
Use it in code
Deploy your container to Cloud Run as a serverless service.
gcloud run deploy rag-api \
--image europe-west1-docker.pkg.dev/my-proj/apps/rag-api:abc123 \
--region europe-west1 --allow-unauthenticated \
--set-secrets ANTHROPIC_API_KEY=anthropic-api-key:latest \
--min-instances 0 --max-instances 5 --timeout 300 --concurrency 40Deploy your RAG API serverless, measure cold-start TTFT, and decide whether to keep one minimum instance.
Common mistakes
- Agent runs that exceed the platform timeout.
- Opening a new database connection on every invocation.