The idea
vLLM is a high-throughput inference engine for open models. It combines PagedAttention, continuous batching, prefix caching, quantization support and tensor parallelism across GPUs, and exposes an OpenAI-compatible HTTP API.
Because the API is OpenAI-compatible, your app can switch between a hosted model and your vLLM server by changing a base URL. You choose the model, its precision and the max context length, which together decide GPU memory needs.
Alternatives include SGLang, TensorRT-LLM and Hugging Face TGI; managed platforms run these for you.
It is the standard way to self-host open models efficiently, central to the GPU vs API decision.
Words to know
- Tensor parallelismSplitting one model across several GPUs.
- max-model-lenThe context length the server allocates for.
Try it without code
Rent a GPU for an hour
- Start a cloud GPU instance with 24 GB or more.
- Install vLLM and serve a 7B instruct model.
- Point a chat client at its /v1 endpoint.
Use it in code
Serve a model and call it with the OpenAI client.
vllm serve Qwen/Qwen2.5-7B-Instruct --max-model-len 8192 --gpu-memory-utilization 0.9
# in another shell
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "Qwen/Qwen2.5-7B-Instruct",
"messages": [{"role": "user", "content": "Say hello in Irish."}],
"max_tokens": 50 }'Serve a 7B model with vLLM, run your load test, and calculate cost per million output tokens from the GPU hourly price.
Common mistakes
- Setting max-model-len far above what you need, wasting KV memory.
- No auth on the endpoint.