Full fine-tuning of multi-billion-parameter models is expensive and awkward when you need many task variants. Parameter-Efficient Fine-Tuning (PEFT)—especially LoRA—trains small adapter matrices while freezing the base. This is the default industrial recipe for open-weight LLM customization.
Learning Objectives
By the end of this lesson, students should be able to:
- Define PEFT and motivate adapters over full FT.
- Explain LoRA: low-rank updates ΔW ≈ BA.
- Configure a LoRA run with Hugging Face PEFT.
- Compare LoRA, QLoRA, and prompt/prefix tuning conceptually.
- Discuss multi-adapter serving and merge strategies.
- List failure modes: too-small rank, wrong target modules.
PEFT methods adapt a large pretrained model by training only a small set of additional (or selected) parameters—adapters, low-rank matrices, or soft prompts—while keeping most base weights frozen. LoRA injects trainable low-rank matrices into linear layers so the effective update is W' = W + BA with rank r ≪ d.
Why Low Rank?
Task adaptation often lies in a low-dimensional subspace of weight space. Training full d×d updates is wasteful; two thin matrices B (d×r) and A (r×d) can express useful shifts with far fewer parameters and smaller checkpoints.
| Method | What you train | Typical use |
|---|---|---|
| LoRA | Low-rank A, B on selected linears | General SFT / domain adapters |
| QLoRA | LoRA on a quantized frozen base | Consumer-GPU fine-tunes |
| Adapters (Houlsby-style) | Small bottleneck modules | Multi-task plug-ins |
| Prompt / prefix tuning | Soft prompt vectors | Very light specialization |
LoRA with Hugging Face PEFT
Engineering Trade-offs
Strengths
- Small artifacts; many adapters per base.
- Cheaper train & store than full FT.
- Easier rollback / A-B of behaviors.
Limits
- May underfit huge distribution shifts.
- Hyperparams (r, targets) matter.
- Merge/serve complexity in fleets.
Train
- Freeze base; train A, B.
- Often with 4/8-bit bases (QLoRA).
Serve
- Load base once; hot-swap adapters.
- Or merge LoRA into W for single-tenant speed.
“LoRA always matches full fine-tuning quality.” Often it is close for instruction/style shifts, but large capability changes or some embedding-level shifts may still need fuller updates or better data—not just bigger r.
Knowledge Check
- Short Answer: What does LoRA approximate? Answer: A low-rank update ΔW ≈ BA added to frozen weights W.
- True/False: PEFT trains every parameter of a 70B model. Answer: False—only a small adapter subset.
- Multiple Choice: QLoRA typically means: (a) LoRA on a quantized base, (b) deleting attention, (c) only CPU k-means, (d) no tokenizer. Answer: (a).
- Short Answer: Name two common LoRA target modules. Answer: q_proj and v_proj (also k/o and MLP linears often).
- True/False: Adapter checkpoints are usually much smaller than full model checkpoints. Answer: True.
- Multiple Choice: A reason to merge LoRA into W is: (a) slower inference always, (b) simpler single-adapter serving, (c) removing the vocabulary, (d) banning GPUs. Answer: (b).
- Short Answer: What hyperparameter is the LoRA rank? Answer: r—the inner dimension of A and B.
- True/False: Prompt tuning updates all transformer weights. Answer: False—it mainly trains soft prompt parameters.
- Multiple Choice: If LoRA underfits a hard domain shift, try: (a) only smaller r always, (b) better data / higher r / more targets / fuller FT, (c) deleting evals, (d) random labels. Answer: (b).
- Short Answer: Why is PEFT popular for multi-tenant LLM products? Answer: One shared base + many small adapters is cheaper to train and store.
Key Takeaways
- PEFT adapts LLMs by training tiny parameter sets.
- LoRA/QLoRA are the workhorse open-weight recipes.
- Serve via adapter swap or merge depending on traffic.
- Next: RLHF aligns models with human preferences.
Lab: Run a tiny LoRA SFT on a small instruct set; print trainable parameter counts vs full model.
Discuss: When would you keep adapters separate vs merge them?
Recap: PEFT (especially LoRA) makes LLM specialization affordable. Continue with RLHF.