Module 6.3 built the stack bottom-up: CPU vs GPU → CUDA cores → VRAM → tensor cores → FP16 / BF16 / INT8. Quantization is the engineering discipline that applies those formats in production: shrink models, cut latency, and fit inside VRAM budgets without shipping naive casts.
This capstone closes Volume 06 Deep Learning. Volume 07 moves to architectures—starting with Convolutional Neural Networks—where quantization makes ResNet and YOLO-class models deployable on phones and browsers.
Learning Objectives
By the end of this lesson, students should be able to:
- Define quantization and place PTQ, QAT, and dynamic quantization on a decision tree.
- Execute a post-training static INT8 pipeline with calibration in PyTorch.
- Compare accuracy, size, and latency before and after quantization.
- Select precision (FP16, BF16, INT8, INT4) for training vs serving contexts.
- Document a reproducible quantization checklist for deployment handoff.
The Quantization Decision Tree
| Method | When | Effort | Accuracy Risk |
|---|---|---|---|
| Dynamic INT8 | Quick CPU inference, linear/RNN layers | Low | Low–moderate |
| PTQ (static) | CV models after FP32 training | Medium | Moderate without calibration |
| QAT | INT8-sensitive models (small nets, detection heads) | High | Lowest among INT8 |
| FP16 / BF16 export | GPU serving with tensor cores | Low | Very low |
End-to-End PTQ Workflow (PyTorch)
Train in FP32 (or BF16), calibrate on a representative loader, convert, validate, export.
Quantization-Aware Training (QAT)
QAT inserts fake-quantize nodes during training so the network learns weights robust to rounding. Use when PTQ drops accuracy below SLA—common in detection, segmentation, and small-footprint edge models you will build with CNNs in Volume 07.
Precision Selection Matrix
| Stage | Recommended Precision | Why |
|---|---|---|
| Training (Ampere+) | BF16 mixed precision | Range + tensor cores, minimal scaler fuss |
| Training (older GPU) | FP16 + GradScaler | Hardware support without BF16 |
| GPU inference | FP16 or BF16 | Easy export, tensor core paths |
| CPU / edge inference | INT8 PTQ or QAT | Size and SIMD INT8 kernels |
| LLM serving | Weight-only INT4/INT8 + FP16 activations | VRAM bound; KV cache dominates |
Capstone Checklist: Ship-Ready Quantized Model
- Baseline — FP32 (or BF16) metrics on frozen test set from Module 5.1 evaluation habits.
- Target — Max acceptable accuracy drop (e.g., ≤ 0.5% top-1), latency p99, max VRAM/RAM.
- Calibrate — 100+ batches matching production input distribution (resolution, normalization).
- Convert — PTQ first; escalate to QAT if SLA missed.
- Benchmark — Same hardware as production; record throughput and memory.
- Export — TorchScript, ONNX, or TensorRT with documented opset and backend.
- Regression — Golden-file outputs for a fixed input tensor in CI.
Take a model trained in Module 6.1–6.2 (or a provided ResNet). Deliver: (1) FP32 vs INT8 accuracy table, (2) model size on disk, (3) median inference ms on CPU, (4) one-page decision memo: PTQ sufficient or QAT required?
Calibration data must come from train/val distribution only. Tuning scales on test leaks information—same rule as hyperparameter tuning in cross-validation.
INT8 kernels differ across CPU (FBGEMM, QNNPACK), NVIDIA TensorRT, and mobile NPUs. Validate on the deployment target; re-export when hardware changes.
Bridging to Volume 07
CNNs multiply parameters in convolution kernels—prime INT8 candidates. When you train a ResNet in Volume 07, return to this checklist: BF16 training on GPU, INT8 PTQ for mobile, FP16 for low-latency GPU serving. Quantization is not a postscript; it is part of architecture planning.
Knowledge Check
- Short Answer: What is quantization? Answer: Mapping high-precision floats to lower-bit integers/floats via scales for smaller/faster models.
- Short Answer: PTQ vs QAT in one line each. Answer: PTQ quantizes after training; QAT simulates quant during fine-tuning.
- True/False: Calibration should use the test set. Answer: False—use train/val-like data only.
- Multiple Choice: Fastest path to try INT8 on a trained MLP: (a) QAT from scratch, (b) dynamic quant, (c) retrain FP64. Answer: (b).
- Short Answer: What does
prepare()do in static quant? Answer: Inserts observers to collect activation stats for scale computation. - Short Answer: When escalate from PTQ to QAT? Answer: When accuracy drop exceeds SLA after proper calibration.
- Short Answer: Best training precision on H100? Answer: BF16 (or FP8 in advanced stacks); not INT8 from scratch.
- True/False: FP16 export is a form of quantization. Answer: True in the broad sense (precision reduction), though INT8 is the classical meaning.
- Multiple Choice: LLM weight-only INT8 primarily saves: (a) optimizer state, (b) VRAM for weights/KV, (c) epoch count. Answer: (b).
- Short Answer: Name three metrics to compare pre/post quant. Answer: Accuracy, latency, model size/memory (any three).
Key Takeaways
- Quantization connects numeric formats to deployment constraints—not just theory.
- Start with PTQ + calibration; use QAT when accuracy SLAs fail.
- Always benchmark on target hardware with documented baselines.
- Training (BF16/FP16) and serving (INT8/FP16) precision choices differ by design.
- Volume 07: CNNs — architectures where these GPU and quantization skills pay off first.
Capstone deliverable: Teams submit FP32 vs INT8 report with accuracy/latency/size table and written go/no-go for production.
Discussion prompt: Module 6.3 covered eight topics—draw the pipeline from “buy GPU” to “ship INT8 model.” Where did VRAM, tensor cores, and calibration each matter?