Area 04 · Evals · Concept 13 of 20

Human Labeling

People grade a sample of outputs. This is the truth you check your judges against.

11–20 · Production levelPhase 4 · Weeks 15–17≈ 1 hour

The idea

Automated judges are only as good as their agreement with people. Human labeling has domain experts grade a sample of outputs using the same rubric. These labels are the ground truth for calibrating judges and the best signal for subtle quality issues.

Make labeling easy: a simple tool showing input, output, sources and rubric buttons; keyboard shortcuts; a notes field. Have two people label an overlapping subset to measure their agreement; if humans disagree, the rubric needs work.

Label continuously in small batches (say 20 a week from production), not once.

Why it matters

It keeps your automated metrics honest and catches problems no judge was told to look for.

Words to know

  • Inter-annotator agreementHow often two human labellers agree, often measured with Cohen’s kappa.

Try it without code

Label with a colleague

  1. Both label the same 20 outputs pass/fail.
  2. Count disagreements and discuss each.
  3. Update the rubric.

Use it in code

Compare judge labels with human labels, including Cohen’s kappa.

python
# pip install scikit-learn
from sklearn.metrics import cohen_kappa_score, confusion_matrix

human = ["pass", "fail", "pass", "pass", "fail", "pass", "fail", "pass"]
judge = ["pass", "fail", "pass", "fail", "fail", "pass", "pass", "pass"]

agree = sum(h == j for h, j in zip(human, judge)) / len(human)
print(f"agreement {agree:.0%}, kappa {cohen_kappa_score(human, judge):.2f}")
print(confusion_matrix(human, judge, labels=["pass", "fail"]))
PRACTICE · TICK IT AFTER THIS

Label 50 outputs yourself, compute judge agreement, and write down the two biggest disagreement patterns.

Common mistakes

  • Labelling without a written rubric.
  • Using only non-experts for expert domains.

Go deeper