The idea
A rubric breaks "good answer" into named criteria, each with a defined scale and examples of each level. For a support reply: correct (0/1), complete (0–2), follows policy (0/1), tone (0–2). Scores become comparable and explainable.
Write anchors: what a 0, 1 and 2 look like for each criterion, ideally with real examples. Without anchors, people and models drift. Score criteria separately (one judge call per criterion, or one call returning all fields) and report each, not just a total.
Rubrics also align a team. Writing one forces product, domain experts and engineers to agree on what good means.
Rubrics turn vague quality debates into numbers you can track and act on.
Words to know
- AnchorA concrete description or example of a score level.
Try it without code
Write a rubric
- Pick one AI feature.
- Write 3–4 criteria with 0/1/2 anchors.
- Score 5 outputs yourself and see where you hesitate; refine those anchors.
Use it in code
Multi-criterion rubric scored with a Pydantic schema.
from pydantic import BaseModel, Field
import anthropic
client = anthropic.Anthropic()
class Score(BaseModel):
correct: int = Field(description="1 if all facts are right, else 0")
complete: int = Field(description="0 misses the point, 1 partial, 2 fully answers")
policy: int = Field(description="1 if it follows the refund policy, else 0")
reason: str
def score(question, answer, policy_text):
msg = client.messages.create(model="claude-sonnet-5-5", max_tokens=400, temperature=0,
tools=[{"name": "score", "description": "Record rubric scores", "input_schema": Score.model_json_schema()}],
tool_choice={"type": "tool", "name": "score"},
messages=[{"role": "user", "content": f"Policy:\n{policy_text}\nQ: {question}\nA: {answer}\nScore it."}])
return Score.model_validate(msg.content[0].input)Score your golden set with a 3-criterion rubric and report the average per criterion.
Common mistakes
- Only reporting a total, hiding which criterion dropped.
- Criteria that overlap.