← Master Index
Vol. 19 Module 19.1 Lecture

BLEU

Metrics & Benchmarking

How This Lesson Fits the Module & Volume

Classification metrics (accuracy through ROC) need a discrete label. Vol. 18 chat APIs emit strings. BLEU (Bilingual Evaluation Understudy) is the classic automatic metric for that world: n-gram precision against one or more human references, with a brevity penalty so models cannot game short outputs.

You already know precision from Vol. 05 / this module. BLEU is precision moved onto tokens. It was built for machine translation; teams still quote it for paraphrases and constrained generation. It is a weak metric for open chat. Sibling ROUGE flips toward recall for summarization. Later lectures add perplexity, human eval, and public benchmarks.

Learning Objectives

By the end of this lesson, students should be able to:

  • Define BLEU as modified n-gram precision (usually 1–4) times a brevity penalty.
  • Compute sentence- and corpus-level BLEU with sacrebleu (preferred) or NLTK.
  • Explain clipping, multiple references, and why corpus BLEU ≠ mean of sentence BLEUs.
  • Know when BLEU is appropriate (MT, constrained wording) vs misleading (open LLM chat).
  • Contrast BLEU (precision-like) with ROUGE (recall-like) using the P/R vocabulary from earlier lectures.
  • Report tokenizer, lowercase, and sacrebleu signature so scores are comparable.
Definition

BLEU scores a hypothesis against reference translation(s) as the geometric mean of modified n-gram precisions (typically n=1..4), multiplied by a brevity penalty (BP) if the hypothesis is shorter than the reference. “Modified” means n-gram counts are clipped to the maximum count in any reference, so repeating “the the the” cannot inflate precision. BLEU ∈ [0, 1] (often reported ×100). It is not accuracy, not F1, and not a meaning metric.

Ingredients (precision you already know)

PieceRoleClassifier analogue
N-gram precision pnFraction of hyp n-grams that appear in a ref (clipped)Precision (extra tokens = FP)
Geometric meanNeeds all n=1..4 to be non-zero (smoothing helps)Harsh like harmonic F1
Brevity penaltyPunishes hyp shorter than refStops empty/high-precision cheats
Multiple refsA hyp n-gram counts if it matches any ref (clipped to max)Several valid labels

Python: sacrebleu (use this in papers) + NLTK

sacrebleu standardizes tokenization so “BLEU 32” is comparable across papers. NLTK is fine for intuition. Never mix them in one leaderboard.

# pip install sacrebleu nltk import sacrebleu from nltk.translate.bleu_score import sentence_bleu, corpus_bleu, SmoothingFunction refs = ["The cat sat on the mat."] hyp_good = "The cat sat on the mat." hyp_close = "The cat is sitting on the mat." hyp_short = "The cat." hyp_wrong = "Quantum foam evaporates at noon." for name, hyp in [("good", hyp_good), ("close", hyp_close), ("short", hyp_short), ("wrong", hyp_wrong)]: print(name, "sacrebleu", round(sacrebleu.sentence_bleu(hyp, refs).score, 2)) # Corpus BLEU on a tiny set (real eval = hundreds+ sentences) # refs_stream: one list per reference source, each list has N sentences hyps = [hyp_close, hyp_good] refs_stream = [["The cat sat on the mat.", "The cat sat on the mat."]] corpus = sacrebleu.corpus_bleu(hyps, refs_stream) print("corpus", round(corpus.score, 2)) print(corpus) # includes sacrebleu signature for reproducibility smooth = SmoothingFunction().method1 ref_tok = [refs[0].lower().split()] print("nltk sentence", sentence_bleu(ref_tok, hyp_close.lower().split(), smoothing_function=smooth)) print("nltk corpus", corpus_bleu([[r.lower().split()] for r in refs * 2], [h.lower().split() for h in hyps]))

When BLEU Helps a Vol. 18 LLM Wrapper

Appropriate

  • Machine translation
  • Tight paraphrase / localization
  • Constrained templates with gold strings
  • Regression tests vs a frozen ref set

Weak or wrong

  • Open-ended chat / tutoring
  • Code (use unit tests, not BLEU)
  • Faithfulness / citations (hallucination tests)
  • Summaries that may use new wording → ROUGE + human

Always pair with

  • Human spot-check
  • Exact-match on structured fields
  • ROUGE if coverage matters
  • Latency / token cost (later lectures)

Gotchas That Break Leaderboards

PitfallWhat to do
Sentence BLEU averaged ≠ corpus BLEUReport corpus BLEU for systems; sentence BLEU only for debugging
Unigram-only “BLEU-1” looks highStandard is BLEU-4 (geo mean 1–4) unless you say otherwise
Different tokenizerssacrebleu signature; do not compare NLTK to Moses to Hugging Face casually
One reference, many valid phrasingsMultiple refs or accept that BLEU under-rewards synonyms
Tiny test setVariance explodes; use hundreds of examples + CI

A fluent LLM can score worse BLEU than a stodgy phrase-based MT system that copies reference n-grams. That is not a bug in the model; it is BLEU measuring overlap, not meaning. Vol. 05 precision had the same limitation: clean positives ≠ complete positives.

Related Lectures

LectureRole
PrecisionConceptual parent of n-gram precision
Recall / ROUGECoverage side; ROUGE next
ROC CurveLast classification metric before generation
PerplexityLikelihood of text, not overlap
Human EvaluationGold standard when BLEU disagrees with quality
BenchmarksPublic MT / LLM suites that still quote BLEU
Common Misconception

“Higher BLEU means a better assistant.” Only for overlap with that reference set—synonyms and valid rewrites lose. Second: sentence-level BLEU without smoothing is often zero because some pn=0; that does not mean the sentence is garbage. Third: comparing your Hugging Face evaluate BLEU to a paper’s sacrebleu without the signature is invalid. Fourth: BLEU on code or JSON APIs is the wrong tool—use parsers and unit tests. Fifth: brevity penalty is not optional decoration; it is what stops precision-hacking with two-word outputs.

Knowledge Check

  1. Short Answer: What does BLEU stand for? Answer: Bilingual Evaluation Understudy.
  2. True/False: BLEU is primarily n-gram precision plus a brevity penalty. Answer: True.
  3. Multiple Choice: Standard BLEU uses n-grams: (a) only unigrams, (b) typically 1 through 4, (c) only characters. Answer: (b).
  4. Short Answer: Why clip n-gram counts? Answer: To stop repeating a common n-gram from inflating precision.
  5. True/False: Mean of sentence BLEUs is the same as corpus BLEU. Answer: False.
  6. Multiple Choice: Best automatic metric for open chat: (a) BLEU alone, (b) BLEU + human / task success, (c) accuracy of random tokens. Answer: (b).
  7. Short Answer: Which sibling metric is more recall-oriented for summaries? Answer: ROUGE.
  8. True/False: sacrebleu includes a reproducible tokenization signature. Answer: True.
  9. Multiple Choice: Next lecture: (a) ROUGE, (b) Docker, (c) SVM. Answer: (a).
  10. Short Answer: Name one Vol. 18 output type where BLEU is the wrong primary metric. Answer: e.g., open chat, code generation, JSON tool calls (accept unit tests / exact field match / human).

Key Takeaways

  • BLEU = clipped n-gram precision (geo mean) × brevity penalty vs references.
  • It is the generation analogue of classifier precision; synonyms get punished.
  • Report corpus BLEU with sacrebleu signature; do not average sentence BLEUs as the system score.
  • Use for MT and constrained text; not as a lone grade for chat LLMs.
  • Next: ROUGE.
Trainer’s Guide

Lab: Score three hypotheses (copy, fluent synonym, short dump) with sacrebleu. Students predict order before running, then explain why the synonym lost. Optional: add a second reference and watch BLEU jump.

Discussion: Product wants “BLEU > 40 on our support bot.” Rewrite the SLO using task success + ROUGE/human, and keep BLEU only if they have gold translations.

Recap: BLEU grades generation with n-gram precision and brevity penalty. Coverage-oriented overlap next: ROUGE.