Stable Diffusion — Module 17.1

How This Lesson Fits the Module & Volume

Module 16.3’s Stable Diffusion lecture was the catalog: RAIL licenses, 1.x vs 2.x, when to pick SD 1.5 vs closed APIs. This Volume 17 lecture is the wiring diagram. You already have noise, denoising, latents, DDPM, and DDIM. SD is those pieces plus CLIP text and cross-attention, shipped as open weights (Rombach et al., 2022, LDM paper; CompVis 1.4; Runway 1.5).

SDXL next scales the same contract. FLUX later changes the denoiser family. Control, LoRA, DreamBooth, and UIs assume you can point to VAE / text encoder / UNet / scheduler on a graph.

Learning Objectives

By the end of this lesson, students should be able to:

Draw SD 1.x as VAE + CLIP text encoder + conditional UNet + scheduler + CFG.
Explain cross-attention: Q from UNet features, K/V from CLIP token embeddings.
Contrast SD 1.4/1.5 (ε-pred, CLIP ViT-L/14, ~512) with SD 2.x (OpenCLIP, often v-pred, 768 option).
Run txt2img and img2img in diffusers and name every tensor that moves.
Diagnose failures as VAE vs CLIP 77-token vs CFG vs sampler vs checkpoint, not “SD is random.”
Point from this architecture to ControlNet, LoRA, inpaint, and A1111/ComfyUI attachment points.

Definition

Stable Diffusion (1.x line) is a text-conditional latent DDPM: a frozen KL-f8 VAE, a frozen (or lightly tuned) CLIP ViT-L/14 text encoder (77 tokens), and a UNet denoiser with spatial self-attention plus cross-attention to text. Sampling is DDPM/DDIM/DPM-class in scaled latent space; classifier-free guidance steers toward the prompt. SD 1.5 (~860M UNet params, native ~512×512) is the community workhorse. SD 2.x swaps in OpenCLIP ViT-H, often v-prediction, and optional 768 native—same LDM contract, not a different science.

The Four-Module Stack

1. Text

CLIP tokenize → embeddings (cond + null for CFG).

2. Latent

Random z_T or encode+noise (img2img).

3. UNet loop

Scheduler steps; cross-attn to text; CFG mix.

4. Decode

Unscale z₀ → VAE decoder → RGB.

Cross-Attention Is How Language Touches Pixels

Inside UNet blocks at several resolutions, image features become queries Q. CLIP token embeddings (after a learned projection) become K and V. Each spatial location can attend to “red bicycle,” “bakery,” “daylight.” This is why token limit 77 matters: overflow is silently dropped, not magically summarized. It is also why ControlNet later adds extra conv branches rather than stuffing pose into the 77 tokens, and why negative prompts are a second embedding stream into the same attention.

Module	Trainable in base SD 1.5?	Job
VAE encoder/decoder	Frozen at LDM train time	Pixels ↔ 4×H/8×W/8 latent
CLIP text encoder	Frozen (usually)	Prompt → 77 × 768 embeddings
UNet	Yes (the diffusion model)	ε_θ(z_t, t, c)
Scheduler	Not weights—config	DDPM/DDIM/DPM math on ᾱ

1.4 vs 1.5 vs 2.x (Architecture, Not Marketplace)

SD 1.4 / 1.5

CLIP ViT-L/14, ε-prediction.
Native ~512; LAION-2B-en lineage.
1.5 = more steps / slightly cleaner 1.4.
Huge LoRA / ControlNet zoo.

SD 2.0 / 2.1

OpenCLIP ViT-H/14.
Often v-prediction + 768 option.
Different tokenizer → prompts do not port blindly.
Smaller adapter ecosystem than 1.5.

Not this lecture

SDXL: dual encoders, 1024, refiner.
SD3: MMDiT + T5, later catalog.
FLUX: flow transformer.

Txt2img and Img2img in diffusers

Read this as tensor logistics, not as a magic API. Img2img is DDIM/DDPM started from a partially noised encoded latent—the noise lecture’s t_start.

import torch
from diffusers import StableDiffusionPipeline, StableDiffusionImg2ImgPipeline
from PIL import Image

dtype = torch.float16
txt = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", torch_dtype=dtype
).to("cuda")
txt.safety_checker = None  # ops choice; not a legal department

out = txt(
    prompt="a red bicycle parked beside a bakery window, daylight, no text",
    negative_prompt="watermark, extra wheels, unreadable letters",
    num_inference_steps=30,
    guidance_scale=7.5,
    generator=torch.Generator("cuda").manual_seed(0),
).images[0]
out.save("sd15_t2i.png")

i2i = StableDiffusionImg2ImgPipeline(**txt.components)
edited = i2i(
    prompt="the same bakery window at dusk, warm interior lamps, no text",
    image=out.resize((512, 512)),
    strength=0.55,              # fraction of T to noise before denoising
    num_inference_steps=30,
    guidance_scale=6.5,
).images[0]
edited.save("sd15_i2i.png")

CFG, Negative Prompts, and Failure Taxonomy

CFG: ε = ε_uncond + s(ε_cond − ε_uncond). SD 1.5 defaults near s = 7.5. Negative prompts replace or augment the uncond embedding. When a still looks wrong, classify before you download another Civitai checkpoint:

Usually not the UNet

Smeared 8-px type → VAE bandwidth + CLIP.
Prompt ignored after token 77 → text encoder limit.
Washed / crunchy → scaling_factor or wrong prediction_type.

Usually the denoiser / sampler

Burnt contrast → CFG too high.
Layout drift in i2i → strength too high.
Different seed look → stochastic sampler / η > 0.

Safety checkers and RAIL licenses from the Vol. 16 catalog still apply. Open weights are not unrestricted commercial rights. Photoreal face checkpoints remain an ethics and legal landmine even when the UNet math is correct.

Common Misconception

“Stable Diffusion is one model; Automatic1111 is Stable Diffusion; and SD 2.1 prompts work unchanged on 1.5.” SD is a lineage plus thousands of fine-tunes. A1111 / ComfyUI are hosts for the four modules. SD 2.x changed the text encoder and often the prediction type—prompt folklore does not transfer. Read the model card’s prediction_type, scheduler, and license the way you would read a transformer’s tokenizer.

Knowledge Check

Short Answer: Name the four modules in the SD 1.x inference stack. Answer: VAE, CLIP text encoder, UNet, scheduler (plus CFG as a sampling rule).
True/False: This lecture replaces the Vol. 16 SD catalog (licenses, when to pick 1.5 vs APIs). Answer: False—16.3 is product choice; 17.1 is architecture.
Multiple Choice: In UNet cross-attention, CLIP tokens provide: (a) Q only, (b) K and V, (c) the VAE scaling factor. Answer: (b).
Short Answer: What is CLIP’s token window in SD 1.x? Answer: 77 tokens.
True/False: Img2img strength is the fraction of the forward noise schedule applied to the encoded latent before reverse sampling. Answer: True.
Multiple Choice: SD 1.5 native resolution is about: (a) 256, (b) 512, (c) 1024 (that is XL). Answer: (b).
Short Answer: Why do SD 2.x prompts not drop in unchanged on 1.5? Answer: Different text encoder (OpenCLIP vs CLIP ViT-L) and often v- vs ε-prediction.
Short Answer: Give one failure that is primarily the VAE, not the UNet. Answer: Smeared small text / lost high-frequency texture after decode (or similar).
Multiple Choice: Typical SD 1.5 CFG scale is about: (a) 1.0 like FLUX schnell, (b) ~7–8, (c) 50. Answer: (b).
True/False: The scheduler contains millions of learned weights. Answer: False—it is the ᾱ / step math, not a third network.

Key Takeaways

SD 1.x = latent DDPM + CLIP cross-attention + CFG; Vol. 16 told you when to use it, this lecture tells you what it is.
Only the UNet is the diffusion model; VAE and text encoder are frozen context most of the time.
77 tokens, f=8 latents, ε-pred, ~512 native, CFG ~7.5: memorize the contract.
Img2img / inpaint / ControlNet / LoRA all attach to this same loop.
Next: SDXL—dual text encoders, micro-conditioning, native 1024, optional refiner.

Trainer’s Guide

Hands-on idea: Print pipe.unet.config, pipe.vae.config.scaling_factor, tokenizer max length, and scheduler class. Then ablate: CFG 1 vs 7.5 vs 15; strength 0.2 vs 0.8 on the same photo. No new checkpoints.

Discussion prompt: If you could freeze the UNet and only swap VAEs, what user-visible metrics would move (sharpness, text, faces) and what would not (pose, prompt semantics)?

Recap: Stable Diffusion is a CLIP-conditioned latent DDPM with a frozen VAE. Continue with SDXL.