LoRA

How This Lesson Fits the Module & Volume

After SFT as the objective, LoRA is the default PEFT parameterization (Vol. 11 PEFT). Instead of updating W, you train low-rank factors A and B so ΔW ≈ BA. The rest of Module 12.4 variants—AdaLoRA, QLoRA, IA3—extend this idea.

Learning Objectives

By the end of this lesson, students should be able to:

Write the LoRA update W′ = W + (α/r) BA.
Choose rank r, alpha, and target modules.
Configure LoRA with Hugging Face PEFT.
Compare merge-on-serve vs multi-adapter routing.
Diagnose underfitting (r too small) and unstable training.
Position LoRA against full FT on memory and quality.

Definition

LoRA (Low-Rank Adaptation) freezes pretrained weights W ∈ &mathbb;R^d×k and learns two thin matrices B ∈ &mathbb;R^d×r and A ∈ &mathbb;R^r×k with r ≪ min(d, k). The forward pass uses W x + scaling · B(A x). Only A and B (plus optional bias) are trained.

Why Low Rank Works

Task updates often lie in a low-dimensional subspace of weight space. Training full d×k deltas is overkill; rank-r adapters capture most useful directions with orders-of-magnitude fewer parameters and portable checkpoints.

Hyperparameter	Role	Typical start
r (rank)	Adapter capacity	8–64
lora_alpha	Scaling (α/r)	16–64
target_modules	Which linears get LoRA	q_proj, v_proj (+ more)
dropout	Regularize adapters	0.05–0.1
bias	Train biases or not	none / lora_only

PEFT Configuration

from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model, TaskType

model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1")
cfg = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
)
model = get_peft_model(model, cfg)
model.print_trainable_parameters()
# Train with standard SFT Trainer; save adapter with model.save_pretrained(...)
# Optional: model.merge_and_unload() for a single dense checkpoint

Freeze W

Base weights fixed

Inject A, B

On target linears

SFT train

Update adapters only

Serve

Swap / merge adapters

Vs Full Fine-Tuning

Tiny checkpoints.
Multi-tenant adapters.
Less forgetting.

Full FT

Max capacity.
Heavy optimizer RAM.
One big artifact.

Serve tip

Merge for latency.
Keep separate for A/B tasks.
Watch merge numerics.

Strengths and Tradeoffs

Strengths

Industry-standard PEFT for LLMs.
Composable with SFT / DPO stacks.
Easy multi-adapter products.

Tradeoffs

Wrong targets → weak adaptation.
Very low r may underfit hard tasks.
Extra matmuls if not merged.

Common Misconception

“Higher rank is always better.” Past a task-dependent point you pay memory and overfit demos without quality gains. Sweep r with a fixed compute budget rather than maximizing it.

Knowledge Check

Short Answer: Write LoRA’s effective weight update. Answer: ΔW ≈ BA (scaled by α/r); W′ = W + scaling·BA.
True/False: Base W is typically frozen during LoRA training. Answer: True.
Multiple Choice: Rank r controls: (a) tokenizer size, (b) adapter capacity, (c) GPU brand. Answer: (b).
Short Answer: Name a common target module pair. Answer: q_proj and v_proj (or attention/MLP linears).
True/False: LoRA adapters are usually much smaller than full checkpoints. Answer: True.
Multiple Choice: Hugging Face library for LoRA: (a) peft, (b) pygame, (c) eslint. Answer: (a).
Short Answer: Why merge adapters at serve time? Answer: Fold BA into W to avoid extra matmuls / simplify deploy.
True/False: LoRA replaces the need for SFT data. Answer: False—it is a parameterization.
Multiple Choice: Compared to full FT, LoRA usually uses: (a) far fewer trainable params, (b) more, (c) infinite. Answer: (a).
Short Answer: Which method adapts rank during training next? Answer: AdaLoRA.

Key Takeaways

LoRA = frozen W + trainable low-rank BA.
Tune r, alpha, and target modules for the task.
Default PEFT choice for LLM SFT.
Enables small, swappable task packs versus full FT.
Next: AdaLoRA.

Trainer’s Guide

Hands-on idea: Train r=4 vs r=64 on the same SFT set; plot trainable params vs eval loss.

Discussion prompt: For attention-only vs all-linear targets, when is the extra cost worth it?

Recap: LoRA adapts LLMs with low-rank updates while freezing the base. Continue with AdaLoRA.