The idea
Automated judges are only as good as their agreement with people. Human labeling has domain experts grade a sample of outputs using the same rubric. These labels are the ground truth for calibrating judges and the best signal for subtle quality issues.
Make labeling easy: a simple tool showing input, output, sources and rubric buttons; keyboard shortcuts; a notes field. Have two people label an overlapping subset to measure their agreement; if humans disagree, the rubric needs work.
Label continuously in small batches (say 20 a week from production), not once.
It keeps your automated metrics honest and catches problems no judge was told to look for.
Words to know
- Inter-annotator agreementHow often two human labellers agree, often measured with Cohen’s kappa.
Try it without code
Label with a colleague
- Both label the same 20 outputs pass/fail.
- Count disagreements and discuss each.
- Update the rubric.
Use it in code
Compare judge labels with human labels, including Cohen’s kappa.
# pip install scikit-learn
from sklearn.metrics import cohen_kappa_score, confusion_matrix
human = ["pass", "fail", "pass", "pass", "fail", "pass", "fail", "pass"]
judge = ["pass", "fail", "pass", "fail", "fail", "pass", "pass", "pass"]
agree = sum(h == j for h, j in zip(human, judge)) / len(human)
print(f"agreement {agree:.0%}, kappa {cohen_kappa_score(human, judge):.2f}")
print(confusion_matrix(human, judge, labels=["pass", "fail"]))Label 50 outputs yourself, compute judge agreement, and write down the two biggest disagreement patterns.
Common mistakes
- Labelling without a written rubric.
- Using only non-experts for expert domains.