The idea
To self-host or fine-tune models you need GPUs. Options: hyperscalers (AWS, GCP, Azure) with many regions and integrations but often limited availability; GPU clouds (Lambda, RunPod, CoreWeave and others) with simpler pricing; and serverless GPU platforms (Modal, Replicate) billed per second.
Pick by GPU memory first: the model weights plus KV cache must fit. A 7–8B model in 16-bit needs about 16 GB plus cache; 4-bit fits on 24 GB cards; 70B models need 80 GB cards or several GPUs.
GPUs are expensive when idle. Shut down training boxes when done, use spot instances for interruptible jobs, and track utilisation.
It is where the GPU vs API decision meets real invoices.
Words to know
- VRAMGPU memory; the main constraint for model size.
- Spot instanceDiscounted capacity that can be reclaimed at short notice.
Try it without code
Price three options
- Look up the hourly price of a 24 GB and an 80 GB GPU on two providers.
- Compute the monthly cost if left running.
Use it in code
Serverless GPU with Modal: a vLLM server billed per second.
# pip install modal ; modal setup
import modal
image = modal.Image.debian_slim().pip_install("vllm")
app = modal.App("qwen-server", image=image)
@app.function(gpu="L40S", timeout=600, scaledown_window=300)
@modal.web_server(port=8000, startup_timeout=600)
def serve():
import subprocess
subprocess.Popen(["vllm", "serve", "Qwen/Qwen2.5-7B-Instruct", "--port", "8000"])Serve a 7B model on a rented GPU for one hour, run your load test, and compute cost per million tokens.
Common mistakes
- Leaving GPU instances running overnight.
- Picking a GPU without checking VRAM needs.