ROC Curve — Module 19.1

How This Lesson Fits the Module & Volume

Accuracy, precision, recall, and F1 all assume a frozen threshold (often 0.5 in a Vol. 18 FastAPI). The ROC curve asks a prior question: do the model’s scores rank positives above negatives at all? Vol. 05 introduced ROC-AUC; here it is the ranking metric you compute before you pick an operating point for production.

This is the last classical classification lecture in the module. Next we leave label-vs-prediction matrices for generated text: BLEU and ROUGE. ROC does not apply to open-ended strings unless you first define a scored detector (toxicity, hallucination judge, retrieval relevance).

Learning Objectives

By the end of this lesson, students should be able to:

Define ROC as TPR vs FPR across all thresholds and AUC as area under that curve.
Compute roc_curve and roc_auc_score from predict_proba / decision scores.
Interpret AUC 0.5 / 0.7 / 0.9 and the random diagonal baseline.
Know when PR-AUC beats ROC-AUC on rare positives.
Pick a threshold from the curve (or from precision-recall) on validation only.
State why ROC-AUC is not a substitute for F1 at the shipped threshold.

Definition

A ROC curve (Receiver Operating Characteristic) plots true positive rate TPR = TP / (TP + FN) (recall) against false positive rate FPR = FP / (FP + TN) as the score threshold sweeps from strict to loose. ROC-AUC (AUROC) is the area under that curve: equivalently, the probability that a random positive scores higher than a random negative. AUC is threshold-free and ranking-based. It is not accuracy and not F1.

Axes and Landmarks

Point / shape	Meaning
(0, 0)	Threshold so high that nothing is positive—no FPs, no TPs
(1, 1)	Threshold so low that everything is positive
(0, 1)	Perfect ranking: all positives before any negative
Diagonal TPR = FPR	Random ranking; AUC = 0.5
Curve below diagonal	Worse than random (flip the score)

sklearn: ROC and AUC

Use probabilities or decision scores—not hard 0/1 labels. Hard labels collapse the curve to a single point.

from sklearn.metrics import (
    roc_curve, roc_auc_score, average_precision_score, f1_score
)
import numpy as np

y_true = np.array([0, 0, 0, 0, 1, 1, 1, 1])
y_score = np.array([0.10, 0.40, 0.35, 0.80, 0.45, 0.90, 0.70, 0.60])

fpr, tpr, thr = roc_curve(y_true, y_score)
print("AUC", roc_auc_score(y_true, y_score))
print("PR-AUC (AP)", average_precision_score(y_true, y_score))
print("thresholds", np.round(thr, 2))
print("FPR", np.round(fpr, 2))
print("TPR", np.round(tpr, 2))

# Operating point for the Vol. 18 API: freeze threshold on validation
best_thr, best_f1 = 0.5, -1.0
for t in np.unique(y_score):
    pred = (y_score >= t).astype(int)
    f1 = f1_score(y_true, pred)
    if f1 > best_f1:
        best_thr, best_f1 = t, f1
print("F1-max thr (demo; use val set)", best_thr, "F1", round(best_f1, 3))

# Hard labels destroy ROC information:
print("AUC on 0/1 preds", roc_auc_score(y_true, (y_score >= 0.5).astype(int)))

ROC-AUC vs PR-AUC vs F1

ROC-AUC

Ranking quality, all thresholds
Uses TNs via FPR
Can look strong when negatives dominate
Good for comparing models before a threshold

PR-AUC (average precision)

Precision vs recall curve
Ignores TNs; focuses on the rare class
Preferred for fraud / PII / rare intent
Baseline ≈ positive prevalence

F1 @ threshold

What users actually experience
Must match the FastAPI cutoff
Ship this number + confusion matrix
AUC high + F1 low = wrong threshold or calibration

Production Reading

AUC (rough)	How to talk about it	Still check
~0.50	No ranking skill	Do not ship; debug labels/features
~0.70	Useful ranking; threshold will hurt someone	PR curve + cost matrix
~0.90+	Strong separation on this test slice	Calibration, slices, drift after Vol. 18 deploy

Multiclass: sklearn roc_auc_score(..., multi_class="ovr", average="macro") one-vs-rest. For detectors wrapped around LLMs (hallucination judge, toxicity), ROC still applies to the judge scores, not to the generated prose. BLEU/ROUGE evaluate the prose itself.

Related Lectures

Lecture	Role
Recall	TPR is recall—the ROC y-axis
Precision	Use the PR curve when positives are rare
F1 Score	Pick a point on the curve, then report F1
Accuracy	Single threshold; ROC is the sweep
BLEU	Leave classification; enter generation metrics
Vol. 05 Model Evaluation	First ROC-AUC in the curriculum

Common Misconception

“AUC 0.99 means the API is 99% accurate.” AUC is ranking, not accuracy at 0.5. Second: computing ROC from hard predictions is nearly worthless. Third: high ROC-AUC on imbalance can coexist with terrible precision—read PR-AUC. Fourth: “We will pick the threshold on test to maximize AUC.” AUC does not pick a threshold; and test is locked (Vol. 05). Fifth: ROC does not grade ChatGPT answers unless you built a scored classifier on top.

Knowledge Check

Short Answer: What does ROC plot (y vs x)? Answer: TPR (recall) vs FPR.
True/False: ROC-AUC equals accuracy at threshold 0.5. Answer: False.
Multiple Choice: Random ranking AUC: (a) 0, (b) 0.5, (c) 1.0. Answer: (b).
Short Answer: Write FPR in TP/TN/FP/FN. Answer: FP / (FP + TN).
True/False: You should pass predict_proba (or scores), not hard labels, into roc_auc_score. Answer: True.
Multiple Choice: Rare fraud: prefer alongside ROC: (a) PR-AUC, (b) only accuracy, (c) BLEU. Answer: (a).
Short Answer: Probabilistic meaning of AUC? Answer: P(score(random positive) > score(random negative)).
True/False: A perfect ROC curve passes through (0, 1). Answer: True.
Multiple Choice: Next lecture: (a) BLEU, (b) Redis, (c) PCA. Answer: (a).
Short Answer: Why is AUC high but production F1 low a common incident? Answer: Wrong/untuned threshold or poor calibration; AUC ignores the shipped cutoff.

Key Takeaways

ROC = TPR vs FPR across thresholds; AUC = ranking probability.
Use scores, not hard labels; compare to the 0.5 diagonal.
On rare positives, pair ROC with PR-AUC; ship F1/P/R at a frozen threshold.
ROC judges detectors; it does not judge free-form LLM text.
Next: BLEU.

Trainer’s Guide

Lab: Train a logistic pipeline (Vol. 05) on an imbalanced set. Plot ROC and PR curves. Students must write one sentence: “We ship threshold t = … because …” using validation only, then report test AUC + F1.

Whiteboard: Draw (0,0), (1,1), diagonal, and a “good” curve. Mark where a high-recall Vol. 18 pre-filter should sit vs a high-precision auto-block.

Recap: ROC-AUC measures ranking across thresholds; production still needs a cutoff. Generation metrics next: BLEU.