Area 01 · LLM · Concept 06 of 20

Temperature and Top-p

Sampling knobs. Low values give predictable answers, high values give varied ones.

01–10 · The basePhase 1 · Weeks 3–6≈ 30 min

The idea

The model outputs a probability for every possible next token. Sampling decides which one to pick. Temperature reshapes the distribution before picking: below 1 it sharpens it (the top choice dominates), above 1 it flattens it (unlikely tokens get more chances). At 0 the model nearly always takes the top token.

Top-p (nucleus sampling) cuts the long tail: it keeps only the smallest set of tokens whose probabilities add up to p, such as 0.9, and samples from those. Top-k does the same with a fixed count. These prevent rare, nonsensical tokens without making every reply identical.

For extraction, classification, code and anything you parse, use low temperature. For brainstorming, names and creative drafts, raise it. Change one knob at a time; many providers advise against tuning both.

Why it matters

Wrong sampling settings show up as flaky tests, inconsistent JSON or dull creative output. Evals are only repeatable when you control them.

Words to know

  • Greedy decodingAlways pick the most likely token.
  • Nucleus samplingAnother name for top-p.

Try it without code

Compare five runs

  1. In an API playground, set temperature to 0 and ask for a product name five times.
  2. Set it to 1 and repeat.
  3. Count how many distinct names you got in each setting.
Suggest one name for a coffee shop run by robots. Reply with the name only.

Use it in code

Run the same prompt at different temperatures and count unique answers.

python
import anthropic
client = anthropic.Anthropic()

prompt = "Suggest one name for a coffee shop run by robots. Reply with the name only."
for temp in (0.0, 0.7, 1.0):
    names = set()
    for _ in range(5):
        msg = client.messages.create(model="claude-haiku-5-5", max_tokens=20,
                                     temperature=temp,
                                     messages=[{"role": "user", "content": prompt}])
        names.add(msg.content[0].text.strip())
    print(temp, len(names), names)
PRACTICE · TICK IT AFTER THIS

Pick sampling settings for three features in an app you want to build and write one sentence justifying each.

Common mistakes

  • Expecting temperature 0 to be perfectly deterministic. Small differences can still appear.
  • Raising temperature to "make it smarter". It makes it more random.

Go deeper