Precision — Module 19.1

How This Lesson Fits the Module & Volume

Accuracy counted every correct row. Precision asks a narrower, operational question: of the cases the shipped model flagged as positive, how many were actually positive? That is the metric for false-positive cost—wrong fraud freezes, spam in the inbox, a RAG router sending junk to an expensive LLM.

Vol. 05 introduced precision beside the confusion matrix; here you use it as a product gate after Vol. 18 APIs. Next lecture is recall (missed positives). Together they become F1. In NLP, BLEU is itself a precision-flavored n-gram score—the same idea in token space.

Learning Objectives

By the end of this lesson, students should be able to:

Define precision as TP / (TP + FP) and interpret it in business language.
Compute binary and multiclass (macro / micro / weighted) precision in sklearn.
Choose precision when false positives are expensive relative to misses.
Raise precision by raising the decision threshold—and know the recall cost.
Connect precision to BLEU’s n-gram precision when grading generated text.
Avoid “high precision” as a slogan without stating the class and threshold.

Definition

Precision (positive predictive value) is TP / (TP + FP): among predicted positives, the fraction that are true. If precision is 0.80, one in five alerts is a false alarm. Precision is silent about false negatives—a model that almost never predicts positive can look extremely precise while catching almost no real cases. That silence is why recall is the sibling metric.

Read Precision Off the Matrix

Question	Formula	Who cares
Were my alerts real?	Precision = TP / (TP + FP)	Ops, legal, users hit by false alarms
Did I catch the real cases?	Recall = TP / (TP + FN)	Safety, fraud loss, missed disease
Overall correct rows?	Accuracy = (TP + TN) / all	Balanced, symmetric-cost tasks

sklearn: Binary Precision

Work a tiny matrix by hand, then confirm with sklearn. Never call precision_score without knowing which label is pos_label.

from sklearn.metrics import (
    precision_score, confusion_matrix, classification_report, precision_recall_curve
)
import numpy as np

y_true = np.array([1, 1, 0, 0, 0, 1, 0, 0])
y_pred = np.array([1, 1, 1, 0, 0, 0, 0, 0])
# TP=2, FP=1, FN=1, TN=4  →  precision = 2/3 ≈ 0.667

print("confusion\n", confusion_matrix(y_true, y_pred))
print("precision", precision_score(y_true, y_pred, pos_label=1))
print(classification_report(y_true, y_pred, digits=3))

# Threshold: scores from a Vol. 18 FastAPI risk endpoint
y_score = np.array([0.92, 0.81, 0.70, 0.20, 0.15, 0.45, 0.10, 0.05])
prec, rec, thr = precision_recall_curve(y_true, y_score)
# Pick the lowest threshold that still keeps precision ≥ 0.80 (on validation, not test)
for p, r, t in zip(prec[:-1], rec[:-1], thr):
    if p >= 0.80:
        print("thr", round(float(t), 2), "P", round(float(p), 3), "R", round(float(r), 3))
        break

Macro, Micro, Weighted (Multiclass)

Micro

Pool TP/FP globally
Equals accuracy in single-label multiclass
Dominated by frequent classes

Macro

Unweighted mean of per-class precision
Rare intents count equally
Default when every class is a product promise

Weighted

Macro weighted by support
Looks closer to overall accuracy
Can hide a broken rare class

When Precision Is the Gate

Shipped system (Vol. 18)	Why precision first	Typical miss cost
Spam / abuse auto-block	False block angers real users	Recall: some spam slips through
Auto-refund / chargeback flag	FP burns money and trust	Manual review queue grows
LLM tool-call “high confidence” only	Wrong tool is worse than asking	More clarifications
Medical screening alert to pager	Often recall first—do not default to precision	Missed disease

Raising the FastAPI threshold (0.5 → 0.8) usually raises precision and lowers recall. Tune on validation; lock the threshold; report test precision at that threshold. A precision number without a threshold is incomplete for any probabilistic model.

Precision in Text Metrics

BLEU is modified n-gram precision plus a brevity penalty. A translation that invents extra n-grams is punished the same way a classifier is punished for extra FPs. ROUGE is more recall-oriented (did the summary cover the reference?). Knowing precision vs recall on classifiers makes BLEU vs ROUGE less mysterious.

Related Lectures

Lecture	Role
Accuracy	Overall correctness; often misleading alone
Recall	The FN-side twin of this lecture
F1 Score	Harmonic mean of P and R
ROC Curve	See precision/recall change with threshold
BLEU	N-gram precision for generation
Vol. 05 Model Evaluation	Original P/R introduction

Common Misconception

“Precision 1.0 means a perfect model.” It can mean the model predicted the positive class once and got lucky—or never predicted it (zero_division warnings). Second: sklearn defaults pos_label=1; if your fraud label is "fraud" or 0, you are measuring the wrong class. Third: micro-precision on imbalanced multiclass is not a rare-class guarantee. Fourth: BLEU “precision” is not classifier precision—same word, different objects (n-grams vs labels).

Knowledge Check

Short Answer: Write the precision formula. Answer: TP / (TP + FP).
True/False: Precision penalizes false negatives. Answer: False—FNs do not appear in the formula.
Multiple Choice: TP=3, FP=1, FN=6: precision is: (a) 3/4, (b) 3/9, (c) 3/10. Answer: (a).
Short Answer: What sklearn argument selects which class is “positive”? Answer: pos_label (or average / labels in multiclass).
True/False: Raising the classification threshold usually increases precision and decreases recall. Answer: True (typical for well-ordered scores).
Multiple Choice: Rare-class product promise: prefer (a) micro precision only, (b) macro precision (and per-class), (c) accuracy only. Answer: (b).
Short Answer: Name one Vol. 18-style system where false positives are the expensive error. Answer: e.g., spam auto-block, auto-refund, high-confidence tool calls.
True/False: BLEU is built on n-gram precision (plus brevity penalty). Answer: True.
Multiple Choice: Next lecture: (a) Recall, (b) Docker, (c) PCA. Answer: (a).
Short Answer: Why can precision be 1.0 with terrible product value? Answer: Almost no positive predictions (or one lucky TP) while most real positives are missed.

Key Takeaways

Precision = TP / (TP + FP): trustworthiness of positive alerts.
Use it when false alarms are costly; always name class, threshold, and averaging.
Macro vs micro vs weighted change the story on imbalanced multiclass.
BLEU is precision-like in token space; classifiers still need recall.
Next: Recall.

Trainer’s Guide

Lab: From a probabilistic classifier, plot precision vs threshold. Students pick a threshold that hits precision ≥ 0.85 on validation, then report test precision and recall at that frozen threshold.

Whiteboard: Fraud freeze vs missed fraud. Force the class to say which error is FP vs FN, then which of precision or recall is the primary gate.

Recap: Precision measures how clean your positive predictions are. Continue with Recall.