Model Compression — Module 18.3

How This Lesson Fits the Module & Volume

Quantization is the compression lever you will use weekly. Model compression is the full kit: distillation, pruning, low-rank factorization / LoRA as a deployable delta, sparsity, and smaller architectures. Volume 06 VRAM and Volume 12 inference tricks (KV cache, FlashAttention, speculative decoding) also compress the runtime—not the checkpoint—by cutting activation traffic and extra forward passes.

Edge AI (next) and Module 18.4 tiers depend on this menu: sometimes you compress; sometimes you stay API-only and compress nothing locally.

Learning Objectives

By the end of this lesson, students should be able to:

List the main compression families: quantization, distillation, pruning, low-rank/PEFT, architecture shrink.
Choose a method from a constraint (VRAM, latency, accuracy floor, update frequency).
Explain why unstructured prune can shrink disk but not GPU speed without sparse kernels.
Treat LoRA/QLoRA as compression of trainable (and sometimes served) parameters.
Relate runtime systems (FlashAttention, spec decode) to “compression of work,” not of weights.
Decide when compression is cheaper than buying Tier 3+ hardware or paying the API.

Definition

Model compression is any technique that reduces the storage, memory, or compute of a trained model while keeping task quality within a budget. Classic pillars: quantization (fewer bits), distillation (small student mimics a teacher), pruning (drop weights or heads), low-rank / PEFT (store ΔW = BA instead of full W), and architecture search (fewer layers/heads by design). Serving-time systems from Volume 12 compress work per token (I/O-aware attention, speculative decoding) without rewriting the checkpoint.

The Compression Menu

Method	What shrinks	Risk	Typical GenAI use
Quantization	Bytes/param, often bandwidth	Quality cliff if too few bits	INT4 7B on consumer GPU
Distillation	Student depth/width / vocab tricks	Needs teacher + data + train time	Small rerankers, edge classifiers
Pruning	Parameters (structured > unstructured for speed)	Unstructured ≠ faster without sparse GEMM	Head/layer drop; MoE expert skip
Low-rank / LoRA	Trainable (and optionally served) deltas	Wrong base; rank too low	Vol. 12 LoRA; multi-tenant adapters
Smaller native model	Everything	Capability gap vs teacher/API	Phi/Gemma-class vs 70B
Runtime (FA, spec decode)	Time / activation peak, not disk	Engine lock-in	FlashAttention, spec decode

Pick from Constraints, Not Hype

VRAM-bound (Tier 2)

Quantize first (INT4/INT8)
Then smaller model family
Distill only if quality still misses

Latency-bound QPS

TensorRT / fused kernels / FA
Speculative decoding (Vol. 12)
Structured prune or smaller student

Many tasks, one base

LoRA adapters, not 12 full FT copies
QLoRA if the base barely fits
API-only if adapters still too heavy

Distill a Tiny Student (sketch)

Distillation is extra training, not a flag. The student matches teacher logits (and sometimes hidden states). This is how edge classifiers and small rerankers get born—not how you magically turn 70B into 7B overnight without data.

import torch
import torch.nn.functional as F

def kd_loss(student_logits, teacher_logits, labels, T=2.0, alpha=0.7):
    """Classic Hinton KD: CE to labels + KL(teacher || student) at temperature T."""
    ce = F.cross_entropy(student_logits, labels)
    s = F.log_softmax(student_logits / T, dim=-1)
    t = F.softmax(teacher_logits / T, dim=-1)
    kl = F.kl_div(s, t, reduction="batchmean") * (T * T)
    return alpha * kl + (1.0 - alpha) * ce

# teacher.eval(); student.train()
# with torch.no_grad():
#     t_logits = teacher(x)
# loss = kd_loss(student(x), t_logits, y)
# Combine with quantization: distill FP16 teacher → INT8/INT4 student for Edge AI.
# LoRA is orthogonal: compress the update, not the teacher architecture.

Compress vs Buy vs API

Compress (or runtime-opt) when

You already own a good teacher/base
VRAM/latency SLO fails at current dtype
Edge / air-gap forbids huge checkpoints
Multi-tenant: LoRA deltas << full copies

Do not compress—yet

Tier 1 API-only is cheaper than a distillation program
Unstructured 90% prune with no sparse kernel (disk win, GPU shrug)
Accuracy floor is “match frontier” on a 3060
You have not measured KV/FA before blaming weight size

Related Lectures

Lecture	Why it sits beside compression
Quantization	Highest-ROI compression for LLMs
Vol. 12 LoRA / QLoRA	Low-rank deltas; quantized base train
FlashAttention	Compresses attention I/O, not the file
Speculative decoding	Compresses decode work with a draft model
Vol. 06 VRAM	The constraint you are optimizing
Edge AI	Where compressed artifacts actually run

Common Misconception

“Compression always means a smaller .safetensors on disk.” FlashAttention and paged KV change runtime memory and latency without shrinking the checkpoint. Second: “Pruning 80% of weights makes the GPU 5× faster.” Unstructured sparsity rarely hits dense tensor cores; you need structured prune or sparse kernels. Third: “LoRA is unrelated to compression.” Serving ten LoRAs instead of ten full fine-tunes is one of the highest-leverage compressions in production (Vol. 12 + 17).

Knowledge Check

Short Answer: Name four compression families. Answer: Quantization, distillation, pruning, low-rank/PEFT (also architecture shrink / runtime opts).
True/False: FlashAttention shrinks the on-disk checkpoint. Answer: False—it reduces attention I/O and peak activations at runtime.
Multiple Choice: First lever for a 7B on 12 GB VRAM: (a) random unstructured prune only, (b) quantization (INT4/INT8), (c) FP64. Answer: (b).
Short Answer: Why can unstructured pruning fail to speed up inference? Answer: Dense tensor cores ignore scattered zeros without sparse kernels; disk shrink ≠ FLOP shrink.
True/False: LoRA can be viewed as compressing the trainable (and optionally served) update. Answer: True.
Multiple Choice: Distillation requires: (a) only renaming files, (b) a teacher, data, and a student training loop, (c) InfiniBand. Answer: (b).
Short Answer: How does speculative decoding “compress work”? Answer: A cheap draft proposes tokens; the target verifies batches, cutting target forwards per accepted token.
True/False: If API TCO is lower than a distillation program, Tier 1 can be the correct “compression” strategy. Answer: True (compress ops, not weights).
Multiple Choice: Multi-tenant styles/tasks with one base: (a) 12 full FT copies, (b) LoRA adapters, (c) 12 Redis clusters. Answer: (b).
Short Answer: Name one Vol. 12 runtime technique that helps VRAM besides quantization. Answer: FlashAttention, paged KV, or continuous batching (any one).

Key Takeaways

Compression = quant + distill + prune + low-rank + smaller arch + runtime systems.
Match the lever to VRAM vs latency vs multi-tenant vs edge.
Unstructured prune ≠ free speed; LoRA is real compression; FA/KV are runtime compression.
Sometimes the winning move is API-only (Module 18.4 Tier 1).
Continue with Edge AI—where compressed models leave the datacenter.

Trainer’s Guide

Lab: Give teams a fake SLO (12 GB VRAM, p95 200 ms, accuracy ≥ teacher−2%). They must propose a stack: quant vs distill vs LoRA vs API, with a one-page VRAM budget including KV.

Discussion: Is speculative decoding “model compression”? Force a precise vocabulary: checkpoint vs work vs memory traffic.

Recap: Compression is a kit, not a synonym for INT4. Continue with Edge AI.