Vol. 12 LoRA taught low-rank adapters on LLM linears: freeze W, train BA. LoRA (Diffusion) is the same algebra on a UNet or DiT—usually attention projections inside the denoiser, sometimes the text encoder. After ControlNet (layout), LoRA is how the community ships styles, characters, and products as 10–200 MB .safetensors files.
DreamBooth is the heavier personalization sibling (often fuller weight updates + prior loss). You will load LoRAs in ComfyUI and Automatic1111 constantly. Base models remain SD, SDXL, and FLUX—a LoRA is worthless on the wrong backbone.
Learning Objectives
By the end of this lesson, students should be able to:
- Write the LoRA update W′ = W + (α/r) BA and map it from LLMs to diffusion denoisers.
- Name typical target modules on a UNet vs a FLUX transformer.
- Load, scale, fuse, and unload a LoRA with
diffusers. - Choose rank r, alpha, and trigger words without cargo-culting CivitAI titles.
- Contrast LoRA with DreamBooth and with ControlNet.
- Warn about base-mismatch (1.5 LoRA on XL/FLUX) and license inheritance.
LoRA for diffusion freezes a pretrained denoiser (UNet or transformer) and optional text encoder(s), then trains low-rank factors A, B on selected linear layers so ΔW ≈ (α/r) BA. At inference you add the adapter (scaled) or fuse it into W. The idea is identical to Vol. 12 PEFT LoRA; only the tensors change: to_q / to_k / to_v / to_out on SD UNets, or MMDiT block linears on FLUX—not q_proj on Mistral.
Same Idea, Different Weights
| Axis | Vol. 12 LoRA (LLMs) | Vol. 17 LoRA (diffusion) |
|---|---|---|
| Frozen W | Transformer decoder / MLP | UNet or DiT (+ maybe CLIP/T5) |
| Typical targets | q_proj, v_proj, o_proj, MLP | attn to_q/k/v/out; sometimes conv/FF; FLUX double/single stream linears |
| Artifact | Adapter for a chat model | .safetensors style/character pack |
| Serve trick | Merge or multi-adapter route | Same: fuse for latency; keep separate for A/B looks |
| Trigger | Usually none (just the tuned task) | Often a rare token / phrase in the prompt |
Hyperparameters That Matter
r (rank)
- Capacity: 4–8 light style, 16–64 character
- Too low underfits identity
- Too high memorizes 12 photos
alpha / scale
- Effective strength ~ α/r at train, scale at infer
- A1111/ComfyUI slider 0.6–1.0 typical
- >1.2 often overcooked
Targets
- Attention-only: cheap, common
- + text encoder: better prompt binding
- Wrong base = noise or crash
Load a LoRA in diffusers
Match the checkpoint family. An SD 1.5 LoRA will not magically remap onto SDXL or FLUX. Trigger words belong in the prompt if the trainer used them. cross_attention_kwargs["scale"] or set_adapters weights control mix when several LoRAs are loaded.
LoRA vs DreamBooth vs ControlNet
LoRA wins when
- You need portable style/character packs
- VRAM for full FT is gone
- Multi-tenant: swap adapters per request
- You already have a good base + ControlNet layout
Choose something else when
- True subject identity on 5 photos → DreamBooth (or LoRA-DreamBooth hybrid)
- Pose/edges → ControlNet, not a “pose LoRA” fantasy
- One-off pixel fix → inpaint
- Wrong-base LoRA downloaded from a random dump
Related Lectures
| Lecture | Why it sits beside diffusion LoRA |
|---|---|
| Vol. 12 LoRA | Same BA math on LLM weights |
| Vol. 11 PEFT | Family of parameter-efficient methods |
| DreamBooth | Heavier subject personalization |
| ControlNet | Layout conditioner often stacked with LoRA |
| FLUX / SDXL | Backbones whose linears you adapt |
| ComfyUI | Load LoRA nodes in production graphs |
“A LoRA is a full checkpoint.” It is a delta. Without the matching base (and often the same precision/architecture), it does nothing useful. Second misconception: “Vol. 12 LoRA and diffusion LoRA are unrelated.” They share the PEFT idea; only target modules and file culture differ. Third: higher rank always means a better character—past a point you overfit the training album.
Knowledge Check
- Short Answer: Write LoRA’s effective weight update. Answer: ΔW ≈ (α/r) BA; W′ = W + scaled BA.
- True/False: Diffusion LoRA is a different mathematical idea from Vol. 12 LLM LoRA. Answer: False—same low-rank PEFT; different layers.
- Multiple Choice: Typical SD UNet LoRA targets: (a) tokenizer merges, (b) attention to_q/to_k/to_v/to_out, (c) Redis keys. Answer: (b).
- Short Answer: Why must an SD 1.5 LoRA not be dropped onto FLUX blindly? Answer: Different backbone (UNet vs DiT) and weight names/shapes.
- True/False: Base W is usually frozen while LoRA trains. Answer: True.
- Multiple Choice: Portable diffusion LoRAs are often: (a) multi-GB full UNets, (b) small .safetensors adapters, (c) CSS themes. Answer: (b).
- Short Answer: Name one infer-time strength control. Answer: adapter weight / lora_scale / UI slider (e.g. 0.8).
- True/False: ControlNet replaces the need for style LoRAs. Answer: False—ControlNet is spatial; LoRA is appearance/subject.
- Multiple Choice: Heavier subject FT sibling next: (a) DreamBooth, (b) PCA, (c) Whisper diarization. Answer: (a).
- Short Answer: Why fuse a LoRA at serve time? Answer: Fold BA into W to avoid extra matmuls / simplify deploy.
Key Takeaways
- Diffusion LoRA = Vol. 12 LoRA on UNet/DiT (and maybe text encoder) weights.
- Match base, rank, alpha, trigger, and infer scale.
- Tiny, swappable packs beat full checkpoints for styles.
- Stack with ControlNet; compare with DreamBooth for identity.
- Continue with DreamBooth.
Lab: Load one style LoRA at scales 0.4, 0.8, 1.2 on the same seed. Optionally train a tiny rank-8 vs rank-32 adapter on 8 images and compare identity vs overfit.
Discussion: When should a product merge LoRAs into a private checkpoint vs keep multi-adapter routing like LLM PEFT serving?
Recap: LoRA adapts diffusion denoisers with the same low-rank idea as LLM PEFT. Continue with DreamBooth.