Precision asked whether alerts were real. Recall (sensitivity, true positive rate) asks whether real positives were found. After a Vol. 18 deploy, this is the metric for missed fraud, missed toxicity, missed PII in a scanner, missed relevant passages in RAG.
Vol. 05 already defined recall on the confusion matrix. Module 19.1 treats it as a shipping constraint: a high-recall gate with a human or second-model review for the extra FPs. F1 blends P and R; ROC plots recall (TPR) against FPR. ROUGE is the generation analogue—did the summary cover the reference?
Learning Objectives
By the end of this lesson, students should be able to:
- Define recall as TP / (TP + FN) and as TPR on an ROC plot.
- Compute binary and averaged multiclass recall in sklearn.
- Choose recall when false negatives are the expensive error.
- Lower the decision threshold to raise recall and quantify the precision drop.
- Contrast recall with ROUGE’s coverage of reference n-grams.
- Design a two-stage pipeline: high-recall filter, then high-precision (or human) confirm.
Recall is TP / (TP + FN): among actual positives, the fraction the model found. If recall is 0.90, one in ten real cases was missed. Recall ignores false positives—a model that labels everything positive has recall 1.0 and useless precision. Also called sensitivity or true positive rate (TPR), the y-axis of the ROC curve.
Precision vs Recall (same matrix, different column)
| Predicted + | Predicted − | |
|---|---|---|
| Actual + | TP (helps both P and R) | FN (hurts recall only) |
| Actual − | FP (hurts precision only) | TN (helps accuracy, not P/R) |
Precision looks down the predicted-positive column. Recall looks across the actual-positive row. Accuracy also cares about TN. That is why the three numbers can disagree after you ship.
sklearn: Recall and a High-Recall Threshold
When Recall Is the Gate
Recall-first products
- Fraud / AML first-pass
- Cancer / safety screening
- PII / secret scanners
- Toxicity pre-filter before publish
- RAG retrieval: did we fetch the needed chunk?
Accept the FP load
- Human review queue
- Second model (high precision)
- More expensive LLM only on suspects
- Budget FPs in Vol. 18 Celery workers
Do not use recall alone
- Always-positive dummy has R=1
- Report precision at the recall target
- Or F2 if recall should weigh more than P
Two-Stage Pattern (production)
A common Vol. 18 + Vol. 19 design: stage A is a cheap high-recall classifier or retriever (catch almost everything); stage B is a slower high-precision model, tool, or human. You report system recall (stage A must not drop the true cases) and system precision (stage B cleans FPs). Evaluating only stage B on already-filtered traffic hides recall failures upstream.
| Stage | Metric to protect | Failure mode |
|---|---|---|
| Retriever / cheap filter | Recall @ k (or binary recall) | Relevant doc never reaches the LLM |
| Reranker / LLM judge | Precision of kept items | Users see junk or false blocks |
| End-to-end | Both + latency / cost | Local metric theater |
Recall in NLP Metrics
ROUGE-N recall asks: what fraction of reference n-grams appear in the hypothesis? A short, pretty summary can have high precision overlap and terrible recall—it missed half the facts. BLEU is the opposite emphasis (precision + brevity penalty). Same P vs R tension as classifiers, now on tokens.
Related Lectures
| Lecture | Role |
|---|---|
| Accuracy | Includes TNs; can hide missed rares |
| Precision | FP-side twin |
| F1 Score | Balance P and R in one number |
| ROC Curve | TPR (recall) vs FPR across thresholds |
| ROUGE | Recall-oriented overlap for summaries |
| Hallucination Tests | Coverage is not faithfulness |
“Recall 100% means we are safe.” Only if the label definition matches the real harm (your “fraud” label may miss new typologies). Second: retrieval recall@k is not answer correctness—the LLM can still hallucinate. Third: sklearn recall_score on multiclass without average errors or defaults in ways students miss—set average="macro" or per-class explicitly. Fourth: lowering threshold forever is not a strategy; you must budget the FP queue (Vol. 18 workers + human eval later in this module).
Knowledge Check
- Short Answer: Write the recall formula. Answer: TP / (TP + FN).
- True/False: Recall is also called sensitivity or TPR. Answer: True.
- Multiple Choice: TP=4, FN=1, FP=20: recall is: (a) 4/5, (b) 4/24, (c) 4/25. Answer: (a).
- Short Answer: What dummy policy achieves recall 1.0 on a binary problem? Answer: Predict positive for every example.
- True/False: False positives appear in the recall formula. Answer: False.
- Multiple Choice: Screening / first-pass fraud should usually optimize: (a) precision only, (b) recall (with a plan for FPs), (c) VRAM. Answer: (b).
- Short Answer: On an ROC curve, recall is which axis? Answer: Y-axis / TPR / sensitivity.
- True/False: ROUGE emphasizes coverage of the reference more than BLEU does. Answer: True.
- Multiple Choice: Next lecture: (a) F1 Score, (b) Flask, (c) UMAP. Answer: (a).
- Short Answer: Why evaluate recall on the retriever, not only on the final LLM answer? Answer: If the chunk never retrieved, the LLM cannot use it—end-to-end hides upstream FN.
Key Takeaways
- Recall = TP / (TP + FN): coverage of actual positives (TPR).
- Gate on recall when misses are the expensive error; budget FPs explicitly.
- Two-stage systems: protect recall upstream, precision downstream.
- ROUGE is recall-like for summaries; BLEU is precision-like for translation.
- Next: F1 Score.
Lab: Set a recall target (e.g. 0.95) on validation scores; find the highest threshold that still meets it; report test precision at that point. Compare to a “always positive” dummy.
Discussion: RAG pipeline: where is recall measured—retriever@k, reranker, or final answer? Students must draw the FN at each stage.
Recap: Recall measures how many real positives you catch. Combine it with precision next: F1 Score.