Area 01 · LLM · Concept 18 of 20

Quantization

Store weights in fewer bits (8-bit, 4-bit) to save memory and run faster, at a small quality cost.

11–20 · Production levelPhase 1 · Weeks 3–6≈ 1 hour

The idea

Weights are normally stored as 16-bit numbers. Quantization maps them to 8-bit or 4-bit integers plus a few scale factors. A model shrinks by 2–4 times, fits on smaller GPUs or laptops, and often runs faster because memory bandwidth is the bottleneck.

Quality drops a little, more at lower bit widths and on smaller models. 8-bit is usually close to lossless; good 4-bit methods (GPTQ, AWQ, GGUF Q4 variants) are acceptable for most uses; 2–3 bit gets noticeably worse.

You meet quantization when running local models (Ollama and llama.cpp use GGUF files like Q4_K_M) and when serving open models cheaply with vLLM.

Why it matters

It decides which models you can self-host and on what hardware, which is a large part of the GPU-vs-API cost question.

Words to know

  • GGUFFile format for quantized models used by llama.cpp and Ollama.
  • AWQ / GPTQPopular 4-bit quantization methods for GPU serving.

Try it without code

Run two sizes locally

  1. Install Ollama.
  2. Run a model at its default 4-bit tag and then a higher-precision tag if your machine allows.
  3. Compare speed and answers on the same five prompts.

Use it in code

Pull and compare quantized variants with Ollama.

bash
ollama pull llama3.2:3b            # default 4-bit quantization
ollama pull llama3.2:3b-instruct-fp16
ollama list                        # compare file sizes
time ollama run llama3.2:3b "Explain quantization in two sentences."
time ollama run llama3.2:3b-instruct-fp16 "Explain quantization in two sentences."
PRACTICE · TICK IT AFTER THIS

Make a table of model, precision, file size, tokens per second and your quality score on 5 prompts.

Common mistakes

  • Comparing a quantized small model with an API frontier model and blaming quantization.
  • Going below 4-bit to fit a model you really cannot afford to run.

Go deeper