Area 07 · Deployment · Concept 10 of 20

GPU Instances

Renting GPU machines in the cloud to serve or fine-tune models.

01–10 · The basePhase 5 · Weeks 18–20≈ 1 hour

The idea

To self-host or fine-tune models you need GPUs. Options: hyperscalers (AWS, GCP, Azure) with many regions and integrations but often limited availability; GPU clouds (Lambda, RunPod, CoreWeave and others) with simpler pricing; and serverless GPU platforms (Modal, Replicate) billed per second.

Pick by GPU memory first: the model weights plus KV cache must fit. A 7–8B model in 16-bit needs about 16 GB plus cache; 4-bit fits on 24 GB cards; 70B models need 80 GB cards or several GPUs.

GPUs are expensive when idle. Shut down training boxes when done, use spot instances for interruptible jobs, and track utilisation.

Why it matters

It is where the GPU vs API decision meets real invoices.

Words to know

  • VRAMGPU memory; the main constraint for model size.
  • Spot instanceDiscounted capacity that can be reclaimed at short notice.

Try it without code

Price three options

  1. Look up the hourly price of a 24 GB and an 80 GB GPU on two providers.
  2. Compute the monthly cost if left running.

Use it in code

Serverless GPU with Modal: a vLLM server billed per second.

python
# pip install modal ; modal setup
import modal

image = modal.Image.debian_slim().pip_install("vllm")
app = modal.App("qwen-server", image=image)

@app.function(gpu="L40S", timeout=600, scaledown_window=300)
@modal.web_server(port=8000, startup_timeout=600)
def serve():
    import subprocess
    subprocess.Popen(["vllm", "serve", "Qwen/Qwen2.5-7B-Instruct", "--port", "8000"])
$ modal deploy server.py
PRACTICE · TICK IT AFTER THIS

Serve a 7B model on a rented GPU for one hour, run your load test, and compute cost per million tokens.

Common mistakes

  • Leaving GPU instances running overnight.
  • Picking a GPU without checking VRAM needs.

Go deeper