LoRA (Diffusion) is PEFT: small deltas, many styles. DreamBooth is subject personalization—teach the model a specific dog, product, or person from a handful of photos so prompts like “a photo of [V] backpack on Mars” still look like that backpack. It sits after LoRA because students must not confuse “a watercolor adapter” with “my client’s SKU.”
You still stand on SD / SDXL / FLUX bases. Layout remains ControlNet. After identity is learned, inpainting edits one region without retraining. UIs: ComfyUI, A1111.
Learning Objectives
By the end of this lesson, students should be able to:
- Define DreamBooth as few-shot subject binding with a rare token + class noun.
- Explain prior-preservation loss and why class images fight language drift.
- Contrast DreamBooth (often fuller UNet/text-encoder FT) with LoRA and LoRA-DreamBooth hybrids.
- Choose photo count, resolution, and unique identifier without collapsing the class.
- List failure modes: overfitting, leaking other subjects, unethical likeness use.
- Know when inpaint + LoRA is cheaper than a new DreamBooth run.
DreamBooth (Ruiz et al., 2023) fine-tunes a text-to-image model so a rare identifier (e.g. sks, ohwx, or a made-up token) paired with a class noun (“dog”, “sneaker”) refers to one real subject. Training uses 3–5+ images of that subject plus a prior-preservation term: generate (or cache) class images from the frozen model and keep predicting them, so “a dog” does not become only your dog. Classic DreamBooth updates UNet weights (and often the text encoder); modern recipes often DreamBooth with LoRA to cut VRAM.
The Recipe
Few, varied angles/lighting
“a [ID] [class]”
Regularize with class images
Or LoRA-DreamBooth
Keep [ID] + class in the text
| Piece | Role | Pitfall if skipped |
|---|---|---|
| Rare token [V] | Hook that did not mean much before FT | Common words (“Anna”) steal existing semantics |
| Class noun | Tells the model the category | No class → identity with no transferable context |
| Prior preservation | Keep the rest of the class intact | Language drift: every dog looks like yours |
| Varied photos | Disentangle identity from background | Overfit sofa + lamp into the “subject” |
DreamBooth vs LoRA
DreamBooth (classic)
- Strong identity on tiny sets
- Heavy checkpoint / VRAM
- Prior loss is first-class
- One subject per run typically
LoRA
- Portable styles & weaker ID
- Tiny files, multi-tenant
- No prior loss unless you add it
- Stack many adapters
LoRA-DreamBooth
- DB objective, LoRA params
- Practical default in 2024+
- Still needs [V] + class discipline
- Easier to ship than full UNet
Training sketch (diffusers-style)
Real scripts (HF train_dreambooth.py / train_dreambooth_lora.py) handle accelerators, prior generation, and checkpointing. The fragment below is the idea: same noise-prediction loss on instance prompts, plus a weighted prior loss on class prompts. Do not treat it as a complete trainer.
When to DreamBooth vs Not
DreamBooth (or LoRA-DB)
- Product SKU, pet, or consented likeness
- Need many novel scenes of the same subject
- Few photos, high identity bar
Skip or substitute
- One-off edit → inpaint
- Style only → LoRA
- Pose lock → ControlNet
- No consent / biometric risk → do not train
Related Lectures
| Lecture | Why it sits beside DreamBooth |
|---|---|
| LoRA (Diffusion) | PEFT alternative / hybrid parameterization |
| Vol. 12 LoRA | Same adapter math if you choose LoRA-DB |
| ControlNet | Pose the personalized subject after FT |
| Inpainting | Local edit without a new subject model |
| SDXL / FLUX | Bases people actually DreamBooth today |
“DreamBooth is just LoRA with a celebrity name in the prompt.” Classic DreamBooth updates (much of) the denoiser and uses prior preservation; LoRA is a parameterization. You can combine them, but skipping the rare token + class + prior is how you destroy “a dog.” Second misconception: more steps always equal better identity—past overfit, the sofa is part of the person. Consent and likeness law are part of the method, not an appendix.
Knowledge Check
- Short Answer: What two prompt pieces identify a DreamBooth subject? Answer: A rare token/id and a class noun (e.g. sks sneaker).
- True/False: Prior preservation exists to stop the class concept from collapsing into your subject. Answer: True.
- Multiple Choice: Classic DreamBooth typically: (a) only swaps the VAE, (b) fine-tunes UNet (± text encoder) on few images, (c) trains Whisper. Answer: (b).
- Short Answer: How does LoRA-DreamBooth differ from classic DB? Answer: Same personalization objective, but only low-rank adapters are trained.
- True/False: DreamBooth and diffusion LoRA are always identical. Answer: False—LoRA is PEFT; DB is a subject-FT recipe (often using LoRA now).
- Multiple Choice: Language drift means: (a) every class image looks like your subject, (b) CFG became zero, (c) JSON mode failed. Answer: (a).
- Short Answer: Why vary backgrounds in the 3–5 photos? Answer: So the model learns identity, not the sofa/lighting of the album.
- True/False: Inpaint is usually cheaper for a one-pixel logo fix than a new DreamBooth. Answer: True.
- Multiple Choice: Next lecture after subject FT: (a) Image Inpainting, (b) CBOW, (c) Kubernetes. Answer: (a).
- Short Answer: Name an ethical stop condition. Answer: No consent / biometric or likeness abuse—do not train.
Key Takeaways
- DreamBooth binds a rare token + class to one real subject.
- Prior preservation fights class collapse / language drift.
- LoRA-DreamBooth is the practical VRAM default; classic DB is heavier.
- Not a substitute for ControlNet (pose) or inpaint (local edit).
- Continue with Image Inpainting.
Lab: If GPU allows, LoRA-DreamBooth a mug (5 photos) with and without prior loss; prompt “a mug” vs “sks mug on the moon.” Discuss consent using a fictional celebrity request.
Whiteboard: Instance loss vs prior loss. Compare checkpoint size: full UNet vs LoRA. Arrow to inpaint for “change the logo only.”
Recap: DreamBooth personalizes a subject; LoRA personalizes cheaply; priors keep the class alive. Continue with Image Inpainting.