ROUGE

How This Lesson Fits the Module & Volume

BLEU asked: are the hypothesis n-grams allowed by the reference (precision)? ROUGE (Recall-Oriented Understudy for Gisting Evaluation) asks the summarization question: did the hypothesis cover the reference? That is the same P vs R split you learned on classifiers, now on tokens after a Vol. 18 summarization or RAG API.

ROUGE is the default automatic number on CNN/DailyMail-style summaries and a common regression check for LLM “summarize this ticket.” It still cannot detect fluent hallucination. Next in the module: perplexity, then latency, tokens, hallucination tests, human eval, and benchmarks.

Learning Objectives

By the end of this lesson, students should be able to:

Define ROUGE-N (unigram/bigram overlap) and ROUGE-L (LCS) with P, R, and F.
Compute ROUGE-1/2/L with rouge_score and read all three numbers, not only F.
Choose ROUGE when coverage of a gold summary matters more than BLEU-style precision.
Explain why extractive copy-paste can inflate ROUGE while abstractive paraphrase can look worse.
Pair ROUGE with hallucination tests and human eval for RAG answers.
Map ROUGE recall to classifier recall and ROUGE F to F1.

Definition

ROUGE is a family of overlap metrics between a candidate text and one or more reference texts. ROUGE-N counts overlapping n-grams: recall is (matched n-grams) / (n-grams in the reference); precision is over the candidate; F is their harmonic mean. ROUGE-L uses the longest common subsequence (LCS) instead of fixed n-grams, rewarding in-order overlap even with gaps. ROUGE-Lsum applies LCS per sentence then aggregates—common for multi-sentence summaries. Scores are typically in [0, 1].

ROUGE Variants You Will See on Dashboards

Variant	What overlaps	Use
ROUGE-1	Unigrams	Content-word coverage; most forgiving
ROUGE-2	Bigrams	Local fluency / phrase copy
ROUGE-L	LCS (sequence order)	Sentence-level structure
ROUGE-Lsum	Per-sentence LCS, then combine	Multi-sentence news summaries
ROUGE-S / SU	Skip-bigrams (less common now)	Older papers

Python: rouge_score

Always print precision, recall, and F. A short, pretty summary can have high P and low R—it missed half the facts, same failure mode as a high-precision / low-recall classifier.

# pip install rouge-score
from rouge_score import rouge_scorer

scorer = rouge_scorer.RougeScorer(
    ["rouge1", "rouge2", "rougeL", "rougeLsum"], use_stemmer=True
)

ref = (
    "The battery failed after three hours of heavy use. "
    "Support issued a replacement under warranty."
)
hyp_cover = (
    "The battery failed after three hours. "
    "Support sent a replacement under warranty."
)
hyp_short = "The battery failed."
hyp_hallucinate = (
    "The battery failed after three hours. "
    "The CEO resigned and the factory closed."
)

for name, hyp in [("cover", hyp_cover), ("short", hyp_short), ("halluc", hyp_hallucinate)]:
    print("===", name, "===")
    for k, v in scorer.score(ref, hyp).items():
        print(k, "P", round(v.precision, 3), "R", round(v.recall, 3), "F", round(v.fmeasure, 3))

# Hugging Face evaluate is an alternative wrapper around similar logic:
# import evaluate
# rouge = evaluate.load("rouge")
# rouge.compute(predictions=[hyp_cover], references=[ref])

BLEU vs ROUGE (keep the Vol. 19 vocabulary)

BLEU

Precision-first + brevity penalty
Born for translation
Punishes extra / invented n-grams
Geo mean of n=1..4

Recall-first heritage (still report P/F)
Born for gisting / summaries
Punishes missing reference content
ROUGE-1/2/L as separate scores

Neither

Faithfulness to a source doc
Factual hallucination
User preference / tone
Use human eval + hallucination tests

RAG and Ticket Summaries After Vol. 18

Setup	How to use ROUGE	Do not forget
Gold human summary exists	ROUGE-1/2/L F on a locked test set	Stemming, newlines (L vs Lsum)
RAG answer vs source passage	ROUGE recall vs source is a crude coverage check	High overlap can still be wrong (quotes out of context)
No gold summary	ROUGE is not defined against “the internet”	Write a rubric or use human / LLM-as-judge carefully
CI regression	Fail if ROUGE-L F drops > δ on a frozen file	Prompt/model SKU in the eval card with Vol. 18 image hash

Extractive systems (copy sentences from the article) often win ROUGE against abstractive LLMs that paraphrase correctly. That is the synonym problem again. If the product wants abstractive tone, ROUGE is a floor check, not the optimization target—human evaluation decides.

Related Lectures

Lecture	Role
BLEU	Precision-oriented sibling
Recall / F1	Same math on labels instead of n-grams
Accuracy	Exact-match still used for closed QA fields
Perplexity	Next: likelihood, not overlap
Hallucination Tests	ROUGE cannot certify faithfulness
Human Evaluation / Benchmarks	Gold and public suites

Common Misconception

“ROUGE-L F1 0.45 means the summary is 45% correct.” It means 45% harmonic overlap with that reference’s LCS, not factual correctness. Second: reporting only F hides a coverage failure (low R). Third: comparing rouge-score (stemmed) to a paper that did not stem is invalid. Fourth: high ROUGE against the source document is not a hallucination test—the model can copy a true sentence and invent another. Fifth: ROUGE-L and ROUGE-Lsum are not interchangeable when references have multiple sentences.

Knowledge Check

Short Answer: What does ROUGE stand for? Answer: Recall-Oriented Understudy for Gisting Evaluation.
True/False: ROUGE-N recall is matched n-grams over n-grams in the reference. Answer: True.
Multiple Choice: ROUGE-L is based on: (a) AUC, (b) longest common subsequence, (c) CUDA. Answer: (b).
Short Answer: Why print P, R, and F, not only F? Answer: A short hyp can have high P / low R (missed facts) with a middling F.
True/False: BLEU is more recall-oriented than ROUGE. Answer: False—BLEU is precision-oriented; ROUGE was designed around recall.
Multiple Choice: Fluent hallucination that adds false facts often: (a) is guaranteed to tank ROUGE-1 P, (b) can still score OK ROUGE if it also copies true n-grams, (c) equals AUC 0.5. Answer: (b).
Short Answer: Name one reason abstractive LLMs lose to extractive baselines on ROUGE. Answer: Paraphrases / synonyms do not match reference n-grams.
True/False: ROUGE requires a reference (or a chosen source text to compare against). Answer: True.
Multiple Choice: Next lecture in this module: (a) Perplexity, (b) Flask, (c) t-SNE. Answer: (a).
Short Answer: Which classifier metric is the closest analogue to ROUGE recall? Answer: Recall (coverage of actual positives / reference content).

Key Takeaways

ROUGE measures n-gram / LCS overlap with a reference; report P, R, and F for 1, 2, and L.
It is the recall-oriented twin of BLEU—built for summaries, used on LLM gisting.
Extractive copy inflates ROUGE; paraphrase and hallucination both confuse it.
Use as a CI floor with gold summaries; never as the only RAG faithfulness metric.
Next: Perplexity.

Trainer’s Guide

Lab: One news paragraph, three hyps: extractive sentences, abstractive paraphrase, hallucinated extra fact. Compute ROUGE-1/2/L. Students must say which hyp would fool a ROUGE-only gate.

Discussion: Support-ticket summarizer from Vol. 18 FastAPI. Write an eval card: gold file hash, rouge-score settings (stemmer, L vs Lsum), fail thresholds, plus a weekly human sample.

Recap: ROUGE scores summary coverage via n-grams and LCS. Continue with Perplexity.