Automatic metrics in this module—accuracy, F1, BLEU, ROUGE, perplexity, hallucination tests—are fast and incomplete. Human evaluation is how you score helpfulness, tone, preference, and borderline faithfulness when no gold string exists.
It is also the bridge to Vol. 20: bias and safety harms are often invisible to MMLU-style tests. This lecture covers Likert ratings, pairwise preference, and inter-rater agreement so your human numbers are science, not vibes. Capstone next: benchmarks.
Learning Objectives
By the end of this lesson, students should be able to:
- Design a Likert (ordinal) rubric with anchored scale points.
- Run pairwise (A/B) preference evals and explain when they beat absolute scores.
- Compute and interpret inter-rater agreement (percent, Cohen’s \(\kappa\), Krippendorff’s \(\alpha\)).
- Write rater guidelines that reduce drift and leakage of model identity.
- Know sample-size and bias pitfalls (order effects, leniency, brand halo).
- Combine human scores with automatic gates instead of replacing either.
Human evaluation is the systematic scoring of model outputs by people under a written protocol. Common designs are Likert / rubric ratings (each output gets an ordinal score on one or more axes) and pairwise comparison (raters pick A vs B, sometimes with ties). Inter-rater agreement measures whether independent raters apply the protocol consistently—without it, your “human score” is noise.
Likert and Rubrics
A Likert item is an ordered scale (e.g. 1–5) with anchors: written examples of what 1 vs 3 vs 5 mean for that axis (correctness, completeness, tone, safety). Average Likert scores as ordinal data carefully—report distributions, not only means. Multi-axis rubrics beat a single “quality” number when failures differ (a witty wrong answer vs a dull correct one).
| Axis (example) | 1 — fail | 3 — usable | 5 — strong |
|---|---|---|---|
| Faithfulness | Contradicts context | Mostly supported, minor extras | All claims grounded or abstains |
| Helpfulness | Off-task / unusable | Partial, user must rework | Actionable, complete for the ask |
| Clarity | Incoherent | Understandable with effort | Clear, well structured |
Pairwise Preference
Why pairwise
- Humans are better at “which is better?” than absolute 1–7.
- Natural for model A vs B or prompt variants.
- Can aggregate to Elo / Bradley–Terry rankings.
Protocol musts
- Blind labels (no model names).
- Randomize A/B position (left-side bias).
- Allow tie / both-bad if the rubric says so.
Limits
- Does not say how good in isolation.
- Intransitive prefs if criteria shift.
- Cost scales with pairs, not items.
Inter-Rater Agreement
If two raters disagree constantly, you are measuring rater personality, not the model. Start with percent agreement (easy, inflated by chance). Cohen’s \(\kappa\) corrects for chance on two raters, categorical labels. Krippendorff’s \(\alpha\) handles more raters, missing cells, and ordinal/interval data—often the better production choice. Calibrate on a gold subset; retrain raters if \(\kappa\)/\(\alpha\) collapses after a guideline change.
Humans catch
- Helpfulness, tone, cultural fit
- Subtle hallucination / omission
- Preference between close models
Humans introduce
- Fatigue, order, brand bias
- Cost and slow iteration
- Disagreement if the rubric is vague
Related Lectures
| Lecture | Role |
|---|---|
| Hallucination tests | Auto gate before expensive raters |
| Accuracy / F1 | When a gold label exists, prefer it |
| Benchmarks | Public suites \(\neq\) your users’ prefs |
| Bias (Vol. 20) | Human eval must include fairness/safety axes |
| Prompt evaluation | Human loops on prompt variants |
“We had three teammates try it; everyone liked v2.” That is a demo, not an eval: no blinding, no rubric, no agreement, tiny \(n\). Second: averaging Likert as if it were interval without checking the distribution. Third: forcing a winner in pairwise when both answers are unsafe. Fourth: showing model names or distinctive formatting. Fifth: treating \(\kappa=0.2\) as “raters basically agree.”
Knowledge Check
- Short Answer: What is a Likert rating in this lecture? Answer: An ordinal anchored score (e.g. 1–5) on a defined quality axis.
- True/False: Pairwise comparison is often easier for humans than absolute scores. Answer: True.
- Multiple Choice: Cohen’s \(\kappa\) corrects percent agreement for: (a) chance, (b) TTFT, (c) BPE. Answer: (a).
- Short Answer: Why randomize A/B position? Answer: Left-side / order bias.
- True/False: Inter-rater agreement is optional if you trust senior raters. Answer: False—you must measure consistency.
- Multiple Choice: Krippendorff’s \(\alpha\) is useful when: (a) many raters / ordinal / missing labels, (b) only GPU TPS, (c) only PPL. Answer: (a).
- Short Answer: Name one human-eval bias besides order effects. Answer: Leniency, brand halo, fatigue, or identity leakage (any).
- True/False: Human eval replaces BLEU/F1 whenever gold labels exist. Answer: False—use automatic metrics when they fit; humans for the rest.
- Multiple Choice: Next lecture: (a) Benchmarks, (b) DBSCAN, (c) Flash Attention. Answer: (a).
- Short Answer: Why blind model names? Answer: To avoid brand/halo bias in ratings or prefs.
Key Takeaways
- Use anchored Likert rubrics and/or blinded pairwise prefs.
- Report inter-rater agreement (\(\kappa\) / \(\alpha\)), not only mean scores.
- Humans catch preference and nuance automatic metrics miss.
- Protocol quality (guidelines, blinding, \(n\)) is the eval.
- Next: Benchmarks (Vol. 19 capstone).
Lab: Two teams rate the same 12 answers on a 1–5 faithfulness rubric (blind). Compute percent agreement and \(\kappa\). Then run pairwise A vs B on 8 pairs; discuss ties. Revise anchors where disagreement clustered.
Whiteboard: Automatic metrics \(\to\) hallucination tests \(\to\) human rubric \(\to\) agreement. Arrow out to Vol. 20: a high-helpfulness answer can still be biased or unsafe.
Recap: Human evaluation uses Likert rubrics and pairwise prefs, made trustworthy by inter-rater agreement and blinding. Place those scores on the public map next with Benchmarks.