Volume 18 shipped the stack: SDKs, FastAPI, Docker, GPUs, CUDA pins, and OS / driver compatibility. A model that boots is not a model that is good. Volume 19 is evaluation. This lecture is the bridge: after deployment and hardware, you measure quality of the shipped system against labels, not against VRAM or TFLOPS.
You already met accuracy, the confusion matrix, precision, and recall in Vol. 05 Model Evaluation. Module 19.1 revisits them as production metrics—then extends the same discipline to LLMs with BLEU, ROUGE, perplexity, latency, and human eval. Accuracy is lecture one because it is the number everyone quotes and the number that most often lies.
Learning Objectives
By the end of this lesson, students should be able to:
- Define accuracy from the four confusion-matrix cells and compute it in sklearn.
- Explain why high accuracy can hide a useless classifier on imbalanced labels.
- Read a confusion matrix and know when to leave accuracy for precision, recall, F1, or ROC-AUC.
- Contrast classification accuracy with token-overlap metrics used for LLM text (BLEU / ROUGE).
- Write a one-page eval card for a shipped Vol. 18 service: dataset version, split, metric, slice.
- Place accuracy inside Module 19.1 before threshold curves and generation metrics.
Accuracy is the fraction of predictions that match the ground-truth label: (TP + TN) / (TP + TN + FP + FN). It treats every example equally and every error type equally. It is a valid headline metric only when classes are roughly balanced and false positives and false negatives cost about the same. For a shipped classifier, accuracy is a starting diagnostic—not a shipping certificate.
From Hardware to Honest Numbers
Vol. 18 asked: does the container run, does CUDA load, does the API return tokens? Vol. 19 asks: on a locked test set (or a labeled production sample), how often is the decision correct—and for whom? A 13B model on an A6000 with 99% accuracy on a 99% negative fraud stream is the dummy classifier from Vol. 05 wearing a GPU.
| Cell | Meaning | Counts toward accuracy? |
|---|---|---|
| True positive (TP) | Predicted positive, actually positive | Yes (correct) |
| True negative (TN) | Predicted negative, actually negative | Yes (correct) |
| False positive (FP) | Predicted positive, actually negative | No (error) |
| False negative (FN) | Predicted negative, actually positive | No (error) |
sklearn: Accuracy vs the Dummy
Always print the confusion matrix beside accuracy. The dummy “always majority class” baseline is the number you must beat—not zero.
When Accuracy Is Enough—and When It Is Not
Accuracy can lead
- Balanced classes (roughly 40–60%)
- Symmetric error cost (spam vs not-spam both annoying)
- Quick smoke test after a deploy
- Multiclass with similar support per class
Leave accuracy behind
- Fraud, disease, safety, rare intents
- Asymmetric cost (missed leak vs extra review)
- Thresholded scores—use ROC / PR
- Generated text—use BLEU / ROUGE / human
Still report it
- Stakeholders expect the word
- Pair it with per-class P/R/F1
- Slice by cohort, locale, model SKU
- Never as the only gate
Accuracy Does Not Transfer to LLMs Unchanged
Exact-match accuracy still exists for closed QA, tool-call JSON, and multiple-choice benchmarks. Open-ended chat has no single “correct token string.” That is why this module continues to BLEU (n-gram precision for translation-like generation) and ROUGE (recall-oriented overlap for summaries). Do not report “accuracy 87%” on free-form assistant output unless you defined a strict match rule (exact string, parsed field, or rubric score).
| Task after Vol. 18 deploy | Honest metric family |
|---|---|
| Binary / multiclass API (intent, toxicity, routing) | Accuracy + confusion matrix + P/R/F1 |
| Ranked risk scores (fraud, churn) | ROC-AUC / PR-AUC, then a chosen threshold |
| Translation / constrained paraphrase | BLEU (plus human spot-check) |
| Summarization / RAG answer coverage | ROUGE, then hallucination tests |
| Open chat / coding assistant | Task success, human eval, benchmarks—not raw accuracy |
Attach to the Docker image: dataset snapshot hash, train/val/test policy (Vol. 05), accuracy and dummy baseline, per-class F1, slice metrics, and the threshold used in FastAPI. Hardware tier (Vol. 18.4) goes on the same card so “we quantized INT4” can be compared to quality drop, not just VRAM saved.
Related Lectures
| Lecture | Role |
|---|---|
| Vol. 18 OS / CUDA / drivers | Stack that must boot before you measure |
| Vol. 05 Model Evaluation | Confusion matrix, P/R/F1, test-set discipline |
| Precision / Recall / F1 | Error-type metrics when accuracy lies |
| ROC Curve | Threshold-free ranking quality |
| BLEU / ROUGE | LLM text overlap when exact-match fails |
| Benchmarks / Human Evaluation | Public suites and the gold standard |
“99% accuracy means the model is production-ready.” On 99% negatives, a constant-negative model already scores 99%. Second: “Accuracy on the training set is what we report.” Report locked test or a labeled production sample—Vol. 05 still applies after Kubernetes. Third: “ChatGPT accuracy is 90%.” Without a defined label and match rule, that sentence is marketing. Fourth: “We deployed on H100s so quality is solved.” Hardware is Vol. 18; quality is this volume.
Knowledge Check
- Short Answer: Write the accuracy formula from TP, TN, FP, FN. Answer: (TP + TN) / (TP + TN + FP + FN).
- True/False: Accuracy weights false positives more heavily than false negatives. Answer: False—every error counts equally.
- Multiple Choice: Dummy always-negative on 95% negatives scores about: (a) 5%, (b) 50%, (c) 95%. Answer: (c).
- Short Answer: Which Vol. 05 artifact shows TP/FP/FN/TN? Answer: The confusion matrix.
- True/False: Exact-match accuracy is the default metric for open-ended chat. Answer: False—use overlap, task success, or human eval.
- Multiple Choice: After a Vol. 18 deploy, accuracy is best used as: (a) the only gate, (b) a smoke metric beside P/R/F1 and a dummy baseline, (c) a CUDA version. Answer: (b).
- Short Answer: Name one LLM metric this module uses when strings do not match exactly. Answer: BLEU or ROUGE (or perplexity / human eval).
- True/False: A model with lower accuracy than the majority dummy can still have useful recall on the rare class. Answer: True—it may trade TNs for TPs; inspect P/R.
- Multiple Choice: Next lecture after Accuracy: (a) Precision, (b) TensorRT, (c) DDPM. Answer: (a).
- Short Answer: What Volume 18 lecture does this volume take over from? Answer: OS, CUDA & Driver Compatibility (Vol. 18 capstone).
Key Takeaways
- Vol. 19 starts where Vol. 18 ends: measure shipped models, not just running hardware.
- Accuracy is (correct) / (all); it lies under imbalance and unequal error costs.
- Always pair accuracy with a dummy baseline and a confusion matrix (Vol. 05).
- Exact-match accuracy does not grade free-form LLM text—continue to P/R/F1, ROC, then BLEU/ROUGE.
- Next: Precision.
Lab: Take any binary labels from a Vol. 18 intent/toxicity stub (or sklearn make_classification with weights=[0.95, 0.05]). Report accuracy, dummy accuracy, and confusion matrix. Students must refuse to “ship” on accuracy alone.
Discussion: Product wants “99% accurate assistant.” Rewrite the requirement as a labeled task (exact JSON field vs summary vs chat) and pick the Module 19.1 metric that actually fits.
Recap: Accuracy is the first Vol. 19 metric—simple, famous, and often misleading. Continue with Precision.