Area 04 · Evals · Concept 05 of 20

Rubric Scoring

Grade against written criteria (correct, complete, tone), each on a clear scale.

01–10 · The basePhase 4 · Weeks 15–17≈ 1 hour

The idea

A rubric breaks "good answer" into named criteria, each with a defined scale and examples of each level. For a support reply: correct (0/1), complete (0–2), follows policy (0/1), tone (0–2). Scores become comparable and explainable.

Write anchors: what a 0, 1 and 2 look like for each criterion, ideally with real examples. Without anchors, people and models drift. Score criteria separately (one judge call per criterion, or one call returning all fields) and report each, not just a total.

Rubrics also align a team. Writing one forces product, domain experts and engineers to agree on what good means.

Why it matters

Rubrics turn vague quality debates into numbers you can track and act on.

Words to know

  • AnchorA concrete description or example of a score level.

Try it without code

Write a rubric

  1. Pick one AI feature.
  2. Write 3–4 criteria with 0/1/2 anchors.
  3. Score 5 outputs yourself and see where you hesitate; refine those anchors.

Use it in code

Multi-criterion rubric scored with a Pydantic schema.

python
from pydantic import BaseModel, Field
import anthropic
client = anthropic.Anthropic()

class Score(BaseModel):
    correct: int = Field(description="1 if all facts are right, else 0")
    complete: int = Field(description="0 misses the point, 1 partial, 2 fully answers")
    policy: int = Field(description="1 if it follows the refund policy, else 0")
    reason: str

def score(question, answer, policy_text):
    msg = client.messages.create(model="claude-sonnet-5-5", max_tokens=400, temperature=0,
        tools=[{"name": "score", "description": "Record rubric scores", "input_schema": Score.model_json_schema()}],
        tool_choice={"type": "tool", "name": "score"},
        messages=[{"role": "user", "content": f"Policy:\n{policy_text}\nQ: {question}\nA: {answer}\nScore it."}])
    return Score.model_validate(msg.content[0].input)
PRACTICE · TICK IT AFTER THIS

Score your golden set with a 3-criterion rubric and report the average per criterion.

Common mistakes

  • Only reporting a total, hiding which criterion dropped.
  • Criteria that overlap.

Go deeper