Area 01 · LLM · Concept 19 of 20

Distillation

Train a small ‘student’ model to copy the answers of a big ‘teacher’ model.

11–20 · Production levelPhase 1 · Weeks 3–6≈ 1 hour

The idea

A large model is accurate but slow and expensive. Distillation uses it as a teacher: you run it on many inputs from your task, keep its outputs (sometimes its full probability distributions), and fine-tune a small student model to reproduce them.

For a narrow task the student can get close to the teacher’s quality at a fraction of the cost and latency. This is how many "small but surprisingly good" models are made, and how teams cut costs on high-volume features.

Check the provider’s terms before using a commercial model’s outputs to train another model. Many restrict this. Open-weight teachers usually allow it.

Why it matters

For a high-volume feature with a stable task, distillation is often the endgame of cost optimisation.

Words to know

  • Teacher / studentThe big model that labels, and the small model that learns.
  • Synthetic dataTraining examples generated by a model.

Try it without code

Plan a distillation

  1. Pick one high-volume task (e.g. ticket tagging).
  2. Write how many examples you would generate, which teacher, which student, and how you would compare them.

Use it in code

Step one of distillation: build a training file from teacher outputs.

python
import json, anthropic
client = anthropic.Anthropic()

inputs = [l.strip() for l in open("tickets.txt") if l.strip()]
with open("train.jsonl", "w") as f:
    for text in inputs:
        msg = client.messages.create(model="claude-sonnet-5-5", max_tokens=10, temperature=0,
            messages=[{"role": "user", "content": f"Label as BILLING, BUG or HOWTO. Label only.\n{text}"}])
        f.write(json.dumps({"messages": [
            {"role": "user", "content": text},
            {"role": "assistant", "content": msg.content[0].text.strip()}]}) + "\n")
# Then fine-tune a small open model on train.jsonl (see the LoRA lesson).
PRACTICE · TICK IT AFTER THIS

Generate 300 teacher labels, train a small student with LoRA, and compare accuracy and cost per 1,000 items.

Common mistakes

  • Distilling without checking teacher quality first.
  • Ignoring licence and terms of service.

Go deeper