The idea
The model outputs a probability for every possible next token. Sampling decides which one to pick. Temperature reshapes the distribution before picking: below 1 it sharpens it (the top choice dominates), above 1 it flattens it (unlikely tokens get more chances). At 0 the model nearly always takes the top token.
Top-p (nucleus sampling) cuts the long tail: it keeps only the smallest set of tokens whose probabilities add up to p, such as 0.9, and samples from those. Top-k does the same with a fixed count. These prevent rare, nonsensical tokens without making every reply identical.
For extraction, classification, code and anything you parse, use low temperature. For brainstorming, names and creative drafts, raise it. Change one knob at a time; many providers advise against tuning both.
Wrong sampling settings show up as flaky tests, inconsistent JSON or dull creative output. Evals are only repeatable when you control them.
Words to know
- Greedy decodingAlways pick the most likely token.
- Nucleus samplingAnother name for top-p.
Try it without code
Compare five runs
- In an API playground, set temperature to 0 and ask for a product name five times.
- Set it to 1 and repeat.
- Count how many distinct names you got in each setting.
Use it in code
Run the same prompt at different temperatures and count unique answers.
import anthropic
client = anthropic.Anthropic()
prompt = "Suggest one name for a coffee shop run by robots. Reply with the name only."
for temp in (0.0, 0.7, 1.0):
names = set()
for _ in range(5):
msg = client.messages.create(model="claude-haiku-5-5", max_tokens=20,
temperature=temp,
messages=[{"role": "user", "content": prompt}])
names.add(msg.content[0].text.strip())
print(temp, len(names), names)Pick sampling settings for three features in an app you want to build and write one sentence justifying each.
Common mistakes
- Expecting temperature 0 to be perfectly deterministic. Small differences can still appear.
- Raising temperature to "make it smarter". It makes it more random.