The idea
A large model is accurate but slow and expensive. Distillation uses it as a teacher: you run it on many inputs from your task, keep its outputs (sometimes its full probability distributions), and fine-tune a small student model to reproduce them.
For a narrow task the student can get close to the teacher’s quality at a fraction of the cost and latency. This is how many "small but surprisingly good" models are made, and how teams cut costs on high-volume features.
Check the provider’s terms before using a commercial model’s outputs to train another model. Many restrict this. Open-weight teachers usually allow it.
For a high-volume feature with a stable task, distillation is often the endgame of cost optimisation.
Words to know
- Teacher / studentThe big model that labels, and the small model that learns.
- Synthetic dataTraining examples generated by a model.
Try it without code
Plan a distillation
- Pick one high-volume task (e.g. ticket tagging).
- Write how many examples you would generate, which teacher, which student, and how you would compare them.
Use it in code
Step one of distillation: build a training file from teacher outputs.
import json, anthropic
client = anthropic.Anthropic()
inputs = [l.strip() for l in open("tickets.txt") if l.strip()]
with open("train.jsonl", "w") as f:
for text in inputs:
msg = client.messages.create(model="claude-sonnet-5-5", max_tokens=10, temperature=0,
messages=[{"role": "user", "content": f"Label as BILLING, BUG or HOWTO. Label only.\n{text}"}])
f.write(json.dumps({"messages": [
{"role": "user", "content": text},
{"role": "assistant", "content": msg.content[0].text.strip()}]}) + "\n")
# Then fine-tune a small open model on train.jsonl (see the LoRA lesson).Generate 300 teacher labels, train a small student with LoRA, and compare accuracy and cost per 1,000 items.
Common mistakes
- Distilling without checking teacher quality first.
- Ignoring licence and terms of service.