Benchmarks — Module 19.1

How This Lesson Fits the Module & Volume

This is the Vol. 19 capstone. You now have classification metrics, n-gram overlap, perplexity, latency, token usage, hallucination tests, and human evaluation. Benchmarks are the shared public (and private) task suites people cite when they say a model is “SOTA.”

Treat MMLU, GSM8K, HumanEval, and their cousins as categories of eval—knowledge, math, code—not as a scoreboard to memorize. This lecture teaches how to read, run, and distrust leaderboards. Vol. 20 then asks what these suites almost never measure: bias, fairness, safety, and governance.

Learning Objectives

By the end of this lesson, students should be able to:

Define a benchmark as a fixed task set + protocol + metric, not a vibes ranking.
Map MMLU-, GSM8K-, and HumanEval-style suites to knowledge / math / code categories.
List leaderboard pitfalls: contamination, prompt sensitivity, cherry-picking, leakage.
Design a small internal benchmark that mirrors your product, not only public sets.
Recap Vol. 19 metrics and choose a mix (auto + human + ops) per use case.
Hand off to Vol. 20 safety and ethics when quality scores look fine and harm does not.

Definition

A benchmark is a published or internal evaluation contract: a dataset (or generator), a prompting / decoding protocol, a scoring rule, and a reporting format. Leaderboards rank systems under that contract. Famous public suites (MMLU-style multiple-choice knowledge, GSM8K-style grade-school math, HumanEval-style code pass@k) are templates—not universal IQ tests. Scores are only comparable when the contract matches; this lecture does not invent or quote fake model numbers.

Benchmark Families (Categories, Not a Score Table)

Family (examples)	Skill it probes	Typical metric	Blind spots
MMLU-style	Broad multiple-choice knowledge / exams	Accuracy (often 5-shot)	Memorization, cultural coverage, no citations
GSM8K-style	Multi-step arithmetic word problems	Exact match / accuracy	Contest math $\neq$ messy business math
HumanEval / MBPP-style	Function synthesis from docstrings	pass@k (unit tests)	Repo-scale engineering, security, style
IF / instruction suites	Follow formatting and constraints	Rule checkers + humans	Long-horizon agents
RAG / domain sets	Grounded answering on your corpus	Faithfulness + task success	Public leaderboards rarely include these
Ops overlays	Speed and spend under the same tasks	TTFT/p95, tokens, \$	Omitted from most accuracy boards

When a vendor card says “MMLU / GSM8K / HumanEval,” read it as: knowledge MCQ, school math, small-function code. Ask for the exact snapshot name, shot count, temperature, and whether tools were allowed. Do not copy numbers from memory into your product docs.

Leaderboard Pitfalls

Data contamination

Train/test overlap or web scrape of the test PDF.
Looks like intelligence; is memorization.
Mitigate: canaries, private holdouts, contamination checks.

Protocol games

Different prompts, CoT, self-consistency, tools.
Cherry-picked temperature or majority vote.
Mitigate: publish the exact harness (lm-eval-style).

Wrong target

SOTA on GSM8K, fails your refunds bot.
Ignores latency, \$, hallucination, bias.
Mitigate: internal benchmark + Vol. 19 mix + Vol. 20.

Public benchmarks buy

Shared language across papers and vendors
Regression signal for general capability
A starting harness you can fork

They do not buy

Your domain truth or RAG faithfulness
Safety, bias, or legal fitness (Vol. 20)
p95 latency or cost per successful ticket

A Tiny Internal Harness (Category Sketch)

Use official libraries (e.g. Eleuther lm-eval-harness, OpenAI simple-evals, HumanEval’s pass@k) for public sets. Below is a pattern for a private task file—not a claim about any model’s score.

# Internal benchmark pattern. Do not hardcode fake public-leaderboard scores.
import json
import re

def normalize_answer(s):
    return re.sub(r"\s+", " ", s.strip().lower())

def exact_match(pred, gold):
    return normalize_answer(pred) == normalize_answer(gold)

def pass_at_k(n_correct_samples, n_samples, k):
    # Unbiased pass@k for code-style unit tests (HumanEval family).
    # n_correct_samples = how many of n_samples passed all tests.
    if n_samples < k:
        raise ValueError("need n_samples >= k")
    # Combinatorial form: 1 - C(n-c, k) / C(n, k) when n-c >= k else 1.0
    c, n = n_correct_samples, n_samples
    if n - c < k:
        return 1.0
    num = 1.0
    for i in range(k):
        num *= (n - c - i) / (n - i)
    return 1.0 - num

def run_internal_suite(items, generate_fn):
    # items: [{id, category, prompt, gold?, unit_tests?}]
    rows = []
    for it in items:
        pred = generate_fn(it["prompt"])
        row = {"id": it["id"], "category": it["category"]}
        if "gold" in it:
            row["em"] = exact_match(pred, it["gold"])
        rows.append(row)
    by_cat = {}
    for r in rows:
        by_cat.setdefault(r["category"], []).append(r)
    report = {
        cat: {"n": len(rs), "accuracy": sum(x.get("em", 0) for x in rs) / len(rs)}
        for cat, rs in by_cat.items() if all("em" in x for x in rs)
    }
    return {"by_category": report, "n_total": len(rows)}

# Always log: model id, prompt template, temperature, max_tokens, date, harness commit.
# Overlay Vol. 19: latency p95, token in/out, hallucination flags, optional human Likert.

Vol. 19 Recap → Vol. 20

Vol. 19 tool	Use when
Accuracy / P / R / F1 / ROC	Labeled classification or detection
BLEU / ROUGE	Overlap vs references (MT, summarization)
Perplexity	Intrinsic LM fit (same tokenizer)
Latency / token usage	UX and unit economics (Vol. 12 / 13)
Hallucination tests	Groundedness & RAG faithfulness (Vol. 11.4 / 14)
Human evaluation	Preference, tone, borderline claims
This lecture	Public + internal task suites; leaderboard literacy
Vol. 20 Bias & ethics	Harm, fairness, security—not captured by MMLU-style boards

Related Lectures

Lecture	Role
Human evaluation	When the benchmark has no gold string
Model evaluation (Vol. 5)	Classical train/val/test discipline
Inference optimization / cost estimation	Speed and \$ beside accuracy
Vol. 20 AI Security & Ethics	Next volume: bias, fairness, governance

Common Misconception

“Highest MMLU wins the RFP.” Public MCQ scores are one capability slice, often contaminated, prompt-sensitive, and silent on RAG, latency, cost, hallucination, and bias. Second: comparing pass@1 at $T=0$ to pass@64 at $T=0.8$ as if they were the same code benchmark. Third: treating a vendor blog chart (no harness commit) as reproducible science. Fourth: skipping a private holdout because “the leaderboard already exists.” Fifth: assuming Vol. 19 quality metrics imply Vol. 20 safety.

Knowledge Check

Short Answer: What four parts make a benchmark a contract? Answer: Dataset (or generator), protocol, metric, and reporting format.
True/False: MMLU-, GSM8K-, and HumanEval-style names refer to knowledge, math, and code eval categories. Answer: True.
Multiple Choice: HumanEval-style scoring is typically: (a) pass@k / unit tests, (b) wiki PPL only, (c) TTFT only. Answer: (a).
Short Answer: Name two leaderboard pitfalls. Answer: Any of: contamination, prompt sensitivity, cherry-picking, tool use mismatch, no private holdout.
True/False: This lecture expects you to memorize current SOTA percentages. Answer: False—do not invent or rely on fake/stale scores.
Multiple Choice: GSM8K-style tasks target: (a) grade-school math word problems, (b) image inpainting, (c) KV cache size. Answer: (a).
Short Answer: Why keep an internal benchmark? Answer: Public suites miss your domain, RAG, UX, and cost.
True/False: High benchmark accuracy guarantees low bias and safe behavior. Answer: False—that is Vol. 20 territory.
Multiple Choice: Next volume starts with: (a) Bias (Vol. 20), (b) PCA, (c) ComfyUI. Answer: (a).
Short Answer: Which Vol. 19 pair covers ops overlays on a benchmark? Answer: Latency and token usage (cost).

Key Takeaways

Benchmarks are protocols, not trophies; compare only under the same contract.
MMLU / GSM8K / HumanEval-style suites = knowledge / math / code categories—no fake scores here.
Distrust leaderboards: contamination, prompts, cherry-picks, missing ops and safety.
Vol. 19 mix: classical metrics + PPL + latency/tokens + hallucination + humans + benchmarks.
Next volume: Vol. 20 Bias (safety & ethics).

Trainer’s Guide

Lab: Fork a tiny internal JSONL (10 knowledge EM, 10 math EM, 5 code unit tests). Run two prompt templates; show score swing. Add p95 latency and \$/item. Discuss what a public MMLU screenshot would have missed.

Whiteboard: Vol. 19 stack $\to$ “can it?” Vol. 20 $\to$ “should it / for whom / with what harm?” Draw contamination as a wormhole from train into test.

Recap: Benchmarks are shared task contracts—MMLU-, GSM8K-, and HumanEval-style categories plus your private suite—read with leaderboard skepticism. Vol. 19 ends here; Vol. 20 opens on Bias and the rest of AI security & ethics.