Area 05 · AI System Design · Concept 09 of 20

Inference Serving with vLLM

An open-source server for running open models fast on your own GPUs.

01–10 · The basePhase 6 · Weeks 21–23≈ 2 hours

The idea

vLLM is a high-throughput inference engine for open models. It combines PagedAttention, continuous batching, prefix caching, quantization support and tensor parallelism across GPUs, and exposes an OpenAI-compatible HTTP API.

Because the API is OpenAI-compatible, your app can switch between a hosted model and your vLLM server by changing a base URL. You choose the model, its precision and the max context length, which together decide GPU memory needs.

Alternatives include SGLang, TensorRT-LLM and Hugging Face TGI; managed platforms run these for you.

Why it matters

It is the standard way to self-host open models efficiently, central to the GPU vs API decision.

Words to know

  • Tensor parallelismSplitting one model across several GPUs.
  • max-model-lenThe context length the server allocates for.

Try it without code

Rent a GPU for an hour

  1. Start a cloud GPU instance with 24 GB or more.
  2. Install vLLM and serve a 7B instruct model.
  3. Point a chat client at its /v1 endpoint.

Use it in code

Serve a model and call it with the OpenAI client.

bash
vllm serve Qwen/Qwen2.5-7B-Instruct --max-model-len 8192 --gpu-memory-utilization 0.9

# in another shell
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen2.5-7B-Instruct",
  "messages": [{"role": "user", "content": "Say hello in Irish."}],
  "max_tokens": 50 }'
PRACTICE · TICK IT AFTER THIS

Serve a 7B model with vLLM, run your load test, and calculate cost per million output tokens from the GPU hourly price.

Common mistakes

  • Setting max-model-len far above what you need, wasting KV memory.
  • No auth on the endpoint.

Go deeper