The idea
Weights are normally stored as 16-bit numbers. Quantization maps them to 8-bit or 4-bit integers plus a few scale factors. A model shrinks by 2–4 times, fits on smaller GPUs or laptops, and often runs faster because memory bandwidth is the bottleneck.
Quality drops a little, more at lower bit widths and on smaller models. 8-bit is usually close to lossless; good 4-bit methods (GPTQ, AWQ, GGUF Q4 variants) are acceptable for most uses; 2–3 bit gets noticeably worse.
You meet quantization when running local models (Ollama and llama.cpp use GGUF files like Q4_K_M) and when serving open models cheaply with vLLM.
It decides which models you can self-host and on what hardware, which is a large part of the GPU-vs-API cost question.
Words to know
- GGUFFile format for quantized models used by llama.cpp and Ollama.
- AWQ / GPTQPopular 4-bit quantization methods for GPU serving.
Try it without code
Run two sizes locally
- Install Ollama.
- Run a model at its default 4-bit tag and then a higher-precision tag if your machine allows.
- Compare speed and answers on the same five prompts.
Use it in code
Pull and compare quantized variants with Ollama.
ollama pull llama3.2:3b # default 4-bit quantization
ollama pull llama3.2:3b-instruct-fp16
ollama list # compare file sizes
time ollama run llama3.2:3b "Explain quantization in two sentences."
time ollama run llama3.2:3b-instruct-fp16 "Explain quantization in two sentences."Make a table of model, precision, file size, tokens per second and your quality score on 5 prompts.
Common mistakes
- Comparing a quantized small model with an API frontier model and blaming quantization.
- Going below 4-bit to fit a model you really cannot afford to run.