← Master Index
Vol. 17 Module 17.1 Lecture

Diffusion

Diffusion Foundations

How This Lesson Fits the Module & Volume

Volume 16 was a catalog. You scored still generators—Stable Diffusion, SDXL, FLUX, DALL·E, Midjourney—and closed the volume on Stable Video Diffusion: open image-to-video built on the same denoising idea. You learned which product to pick. You did not yet open the math.

Volume 17 is the engineering textbook. This lecture is the map for Module 17.1: noise (the forward process), denoising (the reverse process), latent space (why SD is cheap enough to run), then the two classic samplers DDPM and DDIM, then the production stacks Stable Diffusion and SDXL. Later lectures add FLUX, ControlNet, LoRA, DreamBooth, in/outpainting, and UIs. Diffusion is one idea with three moving parts: destroy structure with noise, learn to undo that destruction, optionally do it in a compressed latent.

Learning Objectives

By the end of this lesson, students should be able to:

  • Define a diffusion model as a learned reverse of a gradual noising process.
  • Contrast diffusion with GANs and autoregressive generators on training stability and sampling cost.
  • Name the three engineering layers: forward noise, reverse denoiser, optional latent VAE.
  • Sketch a sampling loop: start from Gaussian noise, iterate a noise predictor, decode if latent.
  • Connect Vol. 16 product SKUs (SD, SDXL, SVD, FLUX) to those layers without confusing catalog with mechanism.
  • Implement a toy forward–reverse step in PyTorch that matches the closed-form noising formula.
Definition

A diffusion model is a generative model that (1) defines a forward process which gradually corrupts data with Gaussian noise until the distribution is (approximately) isotropic Gaussian, and (2) trains a neural network to run that process backwards—denoising step by step—so new samples can be drawn by starting from pure noise. In latent diffusion (Stable Diffusion, SDXL, SVD), both processes run in a VAE latent space rather than on pixels.

From Catalog to Mechanism

Image generation in 16.1 named the task. Module 16.3 named the SKUs. Almost every modern still/video generator you used there is some flavor of diffusion or its close cousin (flow matching). The UI slider labeled “steps” is the reverse chain length. The slider labeled “CFG / guidance” is how hard the denoiser listens to text. The checkpoint file is mostly a UNet (or DiT) that predicts noise, plus a text encoder and often a VAE.

Vol. 16 product viewVol. 17 mechanism view
SD 1.5 “open UNet, ~512”KL-f8 VAE + CLIP-conditioned UNet + DDPM/DDIM-class sampler
SDXL “native ~1024, dual text”Same LDM recipe, bigger UNet, two encoders, micro-conditioning, optional refiner
FLUX “transformer, few steps”Rectified-flow / flow-matching cousin; still noise→data, different ODE
SVD “I2V from a still”SD-class latent diffusion with extra temporal layers over a frame sequence

The Three Layers

Forward: noise

  • Markov chain (DDPM) or SDE/ODE.
  • Closed form: jump to any timestep t in one shot.
  • Schedule βt controls how fast structure dies.

Reverse: denoise

  • Network predicts ε, x0, v, or score.
  • UNet or transformer + time embedding.
  • Optional text via cross-attention / CFG.

Latent (optional)

  • VAE encodes pixels → small z.
  • Denoise z, then decode.
  • Why SD fits on a consumer GPU.

Why Diffusion Beat the Previous Defaults

GANs pit a generator against a discriminator; they can look sharp but are notoriously unstable and can mode-collapse. Autoregressive image models (pixel CNNs, some transformer decoders) are stable but sample one token/pixel at a time. Diffusion trains with a simple regression loss—usually MSE on the added noise—and covers modes well because every training example is used at many noise levels. The price is iterative sampling: tens to thousands of network evaluations per image, which is why DDIM, DPM-Solver, Turbo/Lightning distillations, and flow models exist.

Strengths

  • Stable, scalable training (noise-prediction MSE).
  • High sample diversity vs typical GANs.
  • Natural conditioning (text, pose, mask) at the denoiser.
  • Same recipe transfers to audio, video, 3D latents.

Tradeoffs

  • Slow ancestral sampling (classic DDPM ~1000 steps).
  • Guidance can oversaturate or ignore the prior.
  • Latent VAE can smear fine text and faces.
  • Data/license issues travel with the checkpoint.

A Minimal Forward Step in PyTorch

You do not need the full DDPM sampler yet. The identity that unlocks everything is: if x0 is a clean sample and ε ~ N(0, I), then at timestep t

xt = √ᾱt · x0 + √(1 − ᾱt) · ε

That is the entire forward process in one line. Training samples t, draws ε, builds xt, and asks a network to predict ε (or x0). Sampling starts from xT ~ N(0, I) and walks t downward.

import torch def q_sample(x0, t, alphas_cumprod, noise=None): """Closed-form forward noising: x_t from x_0 at integer timesteps t.""" if noise is None: noise = torch.randn_like(x0) # alphas_cumprod: shape [T], values in (0, 1], decreasing in t a_bar = alphas_cumprod[t].view(-1, 1, 1, 1) return a_bar.sqrt() * x0 + (1.0 - a_bar).sqrt() * noise, noise T = 1000 betas = torch.linspace(1e-4, 0.02, T) # Ho et al. linear schedule alphas = 1.0 - betas alphas_cumprod = torch.cumprod(alphas, dim=0) x0 = torch.randn(2, 3, 32, 32) # stand-in for a tiny image batch t = torch.tensor([0, 500]) # nearly clean vs mid-noise xt, eps = q_sample(x0, t, alphas_cumprod) print(xt.shape, eps.shape) # torch.Size([2, 3, 32, 32]) both

Conditioning, Without the UI Mythology

Text does not “paint” pixels. A text encoder—CLIP in SD 1.x/XL, OpenCLIP, later T5—produces token embeddings. The denoiser attends to them with cross-attention (Vol. 10). Classifier-free guidance (CFG) runs the denoiser twice (conditioned and unconditioned) and extrapolates toward the text. ControlNet, IP-Adapter, and masks are extra condition channels on the same reverse process. You will implement those later; here you only need: the generative engine is iterative denoising, and every control signal is an extra input to that engine.

1. Encode

Optional VAE: image → latent z0.

2. Noise

Build zt (training) or sample zT ~ N(0,I) (inference).

3. Denoise

UNet/DiT + text/pose → predicted noise, step toward z0.

4. Decode

VAE decoder: z0 → pixels (or frames, for SVD).

Common Misconception

“Diffusion is just adding noise to an image, and Stable Diffusion is a different algorithm from DDPM.” Adding noise is only the forward process—useful for training and for img2img strength. Generation is the reverse. SD / SDXL / SVD are latent diffusion models whose reverse sampler is a DDPM/DDIM-class (or faster ODE) procedure. FLUX is a flow-matching relative, not a magically non-diffusion generator. Vol. 16 SKU names are wrappers around this loop.

Knowledge Check

  1. Short Answer: Name the two processes that define a diffusion model. Answer: A forward noising process and a learned reverse denoising process.
  2. True/False: Volume 16 already derived the DDPM ELBO; this volume only repeats product names. Answer: False—16 was catalog/capability; 17 is mechanism.
  3. Multiple Choice: Sampling typically starts from: (a) a blank white canvas, (b) isotropic Gaussian noise, (c) the training set mean image. Answer: (b).
  4. Short Answer: Write the closed-form expression for xt in terms of x0, ᾱt, and ε. Answer: xt = √ᾱt x0 + √(1−ᾱt) ε.
  5. True/False: Latent diffusion runs the UNet on full-resolution RGB pixels. Answer: False—it denoises a VAE latent, then decodes.
  6. Multiple Choice: SVD relates to this lecture because: (a) it is a GAN, (b) it is latent diffusion with temporal layers, (c) it replaces noise with optical flow only. Answer: (b).
  7. Short Answer: Give one reason diffusion training is often more stable than GAN training. Answer: It uses a simple noise-prediction (regression) loss instead of an adversarial min-max game.
  8. Short Answer: What does a text prompt actually condition in SD-class models? Answer: The denoiser, via text-encoder embeddings and cross-attention (plus CFG).
  9. Multiple Choice: FLUX, relative to DDPM, is best described as: (a) unrelated JPEG compression, (b) a flow-matching / transformer cousin of diffusion, (c) SD 1.4 with a new UI. Answer: (b).
  10. True/False: The number of inference “steps” is the length of the reverse chain (or ODE discretization), not the training epoch count. Answer: True.

Key Takeaways

  • Diffusion = learned reverse of gradual Gaussian noising; products in Vol. 16 are mostly this loop plus conditioning and a VAE.
  • The three layers to master next are noise, denoising, and latent space—then DDPM/DDIM make them algorithmic.
  • Closed-form q(xt | x0) lets you jump to any noise level in one shot during training.
  • Iterative sampling is the quality/speed tradeoff; later lectures (DDIM, distillations, FLUX) attack that cost.
  • Next: Noise—schedules, SNR, and why the noise is Gaussian.
Trainer’s Guide

Hands-on idea: Take one CIFAR or even a single photo tensor, run q_sample at t = 0, 50, 200, 500, 999, and display the grid. Students should see structure die before they meet UNets.

Discussion prompt: Open the Vol. 16 SD and SVD lectures side by side. Ask: which sliders (steps, guidance, strength, motion bucket) map to forward noise vs reverse denoising vs extra conditioning?

Recap: Diffusion is a noising process you can invert with a neural net, often inside a latent. Continue with Noise.