DALL·E and Midjourney are closed products. Stable Diffusion (SD) is the open-weight event that turned still generation into something you can run, fine-tune, and graph. This lecture is the catalog view: licenses, 1.x vs 2.x vs later lines, ecosystem (LoRA, ControlNet, UIs), and when SD 1.5 is still the right base.
Volume 17 is where you learn the physics: diffusion, DDPM, latent space, ControlNet, ComfyUI, and Automatic1111. Do not confuse this 16.3 product lecture with the Vol. 17 SD deep dive—here you choose SD; there you open the UNet. Next up is SDXL, then FLUX.
Learning Objectives
By the end of this lesson, students should be able to:
- Define Stable Diffusion as latent diffusion with published weights and a huge adapter ecosystem.
- Score SD 1.x/2.x on the 16.3 rubric vs closed APIs.
- Run a minimal
diffuserspipeline and name the ops costs (VRAM, steps, safety checker). - Distinguish base checkpoint vs LoRA vs ControlNet vs UI (ComfyUI / A1111).
- Read RAIL / community licenses instead of assuming “open = unrestricted commercial.”
- Point ahead to Vol. 17 DDPM, ControlNet, and ComfyUI for real control.
Stable Diffusion is a family of latent diffusion text-to-image models released by Stability AI and partners (Rombach et al., 2022 lineage). A VAE encodes the image to a smaller latent; a UNet (or later transformer) denoises that latent conditioned on text (CLIP / OpenCLIP, later T5 on SD3). Classic community workhorses are SD 1.4/1.5 (~512×512 native) and SD 2.x. SDXL and SD3 are later generations—treat them as siblings, not the same checkpoint.
Rubric: Why Open Weights Change the Job
| Axis | Stable Diffusion 1.5-class | Closed APIs (DALL·E / MJ / Imagen) |
|---|---|---|
| Open vs closed | Weights downloadable; you run inference | Vendor GPU only |
| Prompt fidelity | Weaker out of the box; prompt craft + LoRA + ControlNet close the gap | Often stronger literal following |
| Text rendering | Historically poor on 1.x/2.x (CLIP token limits + no true layout) | Better on later closed models; Ideogram still wins type |
| Licensing / commercial | CreativeML Open RAIL-M (1.5) and variants—use restrictions + share-alike-ish duties; read the file | SaaS ToS + plan |
| Latency | Your GPU: steps × resolution × batch; can be sub-2s with tiny distilled UNets or many seconds on 1.5 full steps | Vendor queue; seconds typical for stills |
Minimal diffusers Call
Hugging Face diffusers is the engineering lingua franca. Automatic1111 and ComfyUI wrap the same idea with graphs and extra networks. Vol. 17 will explain samplers; here you only need: load checkpoint, prompt, steps, guidance, save PNG.
Ecosystem Layers (Catalog Now, Depth in Vol. 17)
Checkpoint
- SD 1.5, 2.1, community fine-tunes
- Sets default look + resolution
- License travels with the file
Adapters
- LoRA / DreamBooth (Vol. 17)
- ControlNet pose, depth, canny
- IP-Adapter / style refs
Typical Use Cases vs Failure Modes
Pick SD 1.5-class when
- On-prem, air-gap, or cost-per-image at scale
- You will train LoRAs on a product SKU
- ControlNet must lock pose/layout
- Huge community checkpoint zoo is a feature
Safety checkers shipped with some diffusers pipelines are not a legal department. RAIL licenses restrict certain uses (e.g. illegal content, some biometric misuse). Community fine-tunes may mix licenses—the file on Civitai is not automatically your commercial right.
“Stable Diffusion is one model, and open means anything goes.” SD is a lineage (1.4/1.5, 2.x, XL, 3.x) plus thousands of fine-tunes. Open weights still carry RAIL or other terms. Photoreal checkpoints trained on scraped faces can be a legal and ethical landmine even when the base 1.5 license looks friendly. Read the card, the license, and your counsel—then learn DDPM so you know what you are actually sampling.
Knowledge Check
- Short Answer: What does “latent diffusion” buy vs pixel-space diffusion? Answer: Denoise a smaller VAE latent—cheaper compute, native ~512 for SD 1.5.
- True/False: This lecture replaces Volume 17 DDPM/ControlNet. Answer: False—16.3 is the catalog; Vol. 17 is the mechanism and control stack.
- Multiple Choice: SD 1.5 text rendering is typically: (a) Ideogram-class, (b) historically weak, (c) native SVG. Answer: (b).
- Short Answer: Name two SD ecosystem layers besides the base checkpoint. Answer: LoRA/DreamBooth and ControlNet (also UIs: ComfyUI/A1111).
- True/False: CreativeML Open RAIL-M means unrestricted commercial use of any Civitai fine-tune. Answer: False—read each license; RAIL has use restrictions; fine-tunes may differ.
- Multiple Choice: Best reason to stay on SD vs DALL·E: (a) Discord grids, (b) self-host + ControlNet/LoRA, (c) ChatGPT rewrite. Answer: (b).
- Short Answer: Which Vol. 17 topics should students preview after this lecture? Answer: Diffusion/DDPM, ControlNet, ComfyUI (and SD lecture in 17.1).
- True/False: Prompt fidelity of vanilla SD 1.5 usually beats DALL·E 3 out of the box. Answer: False—closed DALL·E 3 is typically more literal without adapters.
- Multiple Choice: diffusers
StableDiffusionPipelineis: (a) a Midjourney client, (b) HF inference wrapper for SD checkpoints, (c) a video codec. Answer: (b). - Short Answer: When do you graduate from SD 1.5 to SDXL in this module? Answer: When you want native ~1024, dual text encoders, and the XL ecosystem—next lecture.
Key Takeaways
- SD is the open-weight latent-diffusion line that unlocked LoRA, ControlNet, and local UIs.
- 1.5 is still a workhorse; fidelity and text are craft + adapters, not magic.
- RAIL/open ≠ “no lawyer.” License the checkpoint you actually load.
- Vol. 17 teaches DDPM, ControlNet, and ComfyUI—bookmark it now.
- Continue with SDXL.
Lab: Run the diffusers 1.5 snippet (or A1111 if no CUDA in class). Same prompt with and without a negative prompt. Time 15 vs 30 steps qualitatively—no fake FID.
Whiteboard: Stack diagram: prompt → CLIP → UNet in latent → VAE decode. Arrow out to Vol. 17 ControlNet as extra conditioning. Arrow out to SDXL dual encoders.
Recap: Stable Diffusion is the open latent-diffusion catalog entry; Vol. 17 is the textbook on how it denoises. Continue with SDXL.