← Master Index
Vol. 18 Module 18.3 Lecture

Quantization

Hardware & Model Optimization

How This Lesson Fits the Module & Volume

Volume 06 already defined quantization and dtypes (FP16, BF16, INT8). This lecture is the production catalog: PTQ vs QAT, weight-only INT4 (GPTQ/AWQ/GGUF), bitsandbytes NF4 for QLoRA, and TensorRT/ONNX Runtime INT8. Module 18.4 Tiers 2–4 are unusable without this: a 7B FP16 model (~14 GB weights) does not fit an 8–12 GB RTX 3060; INT4 does.

KV cache can stay FP16/BF16 even when weights are INT4—quantizing weights does not automatically quantize KV. FlashAttention still cares about the compute dtype.

Learning Objectives

By the end of this lesson, students should be able to:

  • Estimate VRAM from parameter count × bytes/param for FP16, INT8, and INT4.
  • Distinguish PTQ, QAT, weight-only quant, and KV-cache quant.
  • Load a 4-bit Hugging Face model with bitsandbytes (BitsAndBytesConfig).
  • Place GPTQ / AWQ / GGUF / bitsandbytes on a decision grid.
  • Connect QLoRA (Vol. 12) to NF4 bases + higher-precision adapters.
  • Warn that “INT4” is not free accuracy—evaluate before shipping (Vol. 19 preview).
Definition

Quantization maps high-precision tensors (usually FP16/BF16/FP32 weights and/or activations) onto fewer bits (INT8, INT4, FP8, NF4) using scales—and sometimes zero-points or lookup codes. Weight-only quantization stores W in low bits and dequantizes into FP16/BF16 for matmul; weight+activation (classic INT8) quantizes both sides so tensor cores run integer GEMM. PTQ calibrates after training; QAT trains with fake-quant. LLM serving mostly uses weight-only PTQ (GPTQ, AWQ, GGUF Q4) or bitsandbytes NF4 for training memory.

Bytes per Parameter (the shop-floor table)

Ignore marketing. For a dense LLM, weight VRAM ≈ params × bytes/param, plus KV + CUDA tax from the GPU lecture. A 7B model: FP16 ~14 GB, INT8 ~7 GB, INT4 ~3.5–4.5 GB depending on extras (scales, unused modules).

Format~Bytes/paramTypical use7B weight ballpark
FP324Legacy train; almost never serve~28 GB
FP16 / BF162Quality infer; A100/4090 class (Tier 3)~14 GB
INT8 / FP8~1TensorRT / Hopper; some PTQ~7 GB
INT4 / NF4 / GGUF Q4~0.5Consumer GPUs, llama.cpp (Tier 2)~4 GB
QLoRA NF4 base + FP16 LoRAbase ~0.5 + tiny adaptersVol. 12 QLoRA train (Tier 4)7B on ~10–16 GB

Families You Will Actually Meet

bitsandbytes NF4

  • On-the-fly load; great for QLoRA
  • Not the fastest decode engine
  • Hugging Face BitsAndBytesConfig

GPTQ / AWQ

  • Offline PTQ; fused kernels in vLLM/HF
  • Better serve throughput than bnb
  • Calibrate on a small corpus

GGUF (llama.cpp)

  • CPU + consumer GPU; many Q k-quants
  • Edge / laptop / Ollama culture
  • Different stack than PyTorch CUDA

Load INT4 with bitsandbytes

This is the practical QLoRA / small-VRAM load path. Compute dtype (BF16/FP16) is what GEMMs run after dequant—tie to Vol. 06 BF16 vs FP16. Double quant shrinks the scale tensors. For serving QPS, prefer GPTQ/AWQ or GGUF once the model is frozen.

import torch from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig bnb = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", # QLoRA paper default bnb_4bit_compute_dtype=torch.bfloat16, # Vol. 06 BF16 on Ampere+ bnb_4bit_use_double_quant=True, ) name = "mistralai/Mistral-7B-Instruct-v0.2" tok = AutoTokenizer.from_pretrained(name) model = AutoModelForCausalLM.from_pretrained( name, quantization_config=bnb, device_map="auto", ) # INT8 alternative: BitsAndBytesConfig(load_in_8bit=True) # KV cache is still typically FP16/BF16 — budget it (Vol. 12), not just weight GB. # Train adapters: attach PEFT LoRA → this is QLoRA (Vol. 12.4), Tier 4 hardware.

Weights vs KV vs Activations

Quantize first

  • Frozen dense weights (biggest static pool)
  • Embedding sometimes left higher precision
  • FP8 activations on Hopper when the engine supports it

Do not assume

  • INT4 weights ⇒ INT4 KV (usually false)
  • PTQ ⇒ no eval needed
  • bnb 4-bit ⇒ best tokens/s (often not)
  • Same Q4 across GPTQ vs GGUF vs AWQ numerics

Related Lectures

LectureWhy it sits beside quantization
Vol. 06 QuantizationScale/zero-point theory
Vol. 06 INT8 / FP16 / BF16Dtype ladder
Vol. 12 QLoRANF4 base + LoRA train
Vol. 12 KV CacheStill budget KV after weight quant
TensorRT / ONNXINT8 calibration / ORT quant
18.4 Tier 2Hardware that exists because of INT4
Common Misconception

“INT4 is just FP16 divided by four with identical answers.” You throw away information; good PTQ (AWQ/GPTQ) is surprisingly strong, but you still eval. Second: “Quantization is only for inference.” QLoRA quantizes the frozen base during training. Third: “If weights fit, we are done.” Long context KV (Vol. 12) OOMs 12 GB cards that comfortably hold Q4 7B weights at 2k context.

Knowledge Check

  1. Short Answer: Approximate FP16 vs INT4 weight size for 7B params. Answer: ~14 GB FP16 vs ~3.5–4.5 GB INT4 (+ KV/scratch).
  2. True/False: Weight-only INT4 automatically quantizes the KV cache. Answer: False—KV is often still FP16/BF16.
  3. Multiple Choice: QLoRA uses: (a) FP32 full FT only, (b) NF4/4-bit base + LoRA adapters, (c) CSS minification. Answer: (b).
  4. Short Answer: PTQ vs QAT in one line. Answer: PTQ calibrates after training; QAT trains with fake-quantization.
  5. True/False: bitsandbytes NF4 load is usually the highest tokens/s serving path. Answer: False—GPTQ/AWQ/GGUF engines often decode faster.
  6. Multiple Choice: Best first quant for an RTX 3060 12 GB chat 7B: (a) FP32, (b) INT4/Q4 or INT8, (c) FP64. Answer: (b).
  7. Short Answer: What does bnb_4bit_compute_dtype=torch.bfloat16 control? Answer: The dtype used for compute after dequantizing NF4 weights.
  8. True/False: Volume 06 INT8 theory still applies to TensorRT INT8 calibration. Answer: True.
  9. Multiple Choice: GGUF Q4 is most associated with: (a) llama.cpp / Ollama, (b) TPU JAX pmap only, (c) Redis. Answer: (a).
  10. Short Answer: Why can a Q4 7B still OOM on 12 GB at 32k context? Answer: KV cache (and fragmentation) grow with seq × batch even if weights are small.

Key Takeaways

  • Quantization is how consumer GPUs run 7B–13B locally—Vol. 06 theory, this lecture’s formats.
  • bytes/param × params + KV = the real budget.
  • bnb NF4 → QLoRA train; GPTQ/AWQ/GGUF → serve.
  • Always evaluate quality; INT4 is a trade, not a free lunch.
  • Continue with Model Compression—quant is one tool in a larger kit.
Trainer’s Guide

Lab: On any CUDA box that can hold it, load the same 7B in FP16 (if VRAM allows) vs 8-bit vs 4-bit bnb. Record memory_allocated and a 50-token generation. If no GPU, compute the bytes/param table by hand and map to Module 18.4 tiers.

Discussion: A product wants “GPT-4 quality on a 3060.” Separate (1) what INT4 7B can do, (2) when to stay API-only, (3) when QLoRA on a bigger box beats pretending 4-bit 7B is frontier.

Recap: Quantization shrinks bytes/param so models fit real VRAM; KV still counts. Continue with Model Compression.