DDPM — Module 17.1

How This Lesson Fits the Module & Volume

You now have noise, denoising, and latents. DDPM (Ho, Jain, Abbeel, 2020, Denoising Diffusion Probabilistic Models) is the discrete algorithm that made that story trainable: a T-step Markov chain, a variational bound, and a simplified MSE on ε. Every Stable Diffusion checkpoint still speaks this language even when the UI sampler is DDIM or DPM++.

Vol. 16 never showed Algorithm 1. This lecture does. DDIM next keeps the same trained ε_θ and changes only the reverse chain. FLUX will later replace the discrete VP path with flow matching—still easier if DDPM is in your bones.

Learning Objectives

By the end of this lesson, students should be able to:

State the DDPM forward chain q(x_t|x_t−1) and the closed-form q(x_t|x₀).
Explain that training maximizes a variational lower bound on log p_θ(x₀), then uses L_simple.
Write L_simple = E_x₀,ε,t [ ||ε − ε_θ(x_t, t)||² ] and implement the training step.
Implement ancestral sampling, including the extra Gaussian at t > 0.
Recall the original hyperparameters: T=1000, linear β from 10⁻⁴ to 0.02, UNet + self-attention.
Know what transfers unchanged when you move DDPM from pixels to SD latents.

Definition

A Denoising Diffusion Probabilistic Model (DDPM) is a latent-variable generative model whose approximate posterior is a fixed Gaussian Markov chain that adds noise for T steps, and whose generator is a learned Markov chain that removes noise. The reverse conditionals are Gaussians whose mean is parameterized by a time-conditioned network ε_θ (a UNet in the original paper). Training uses the variational bound on −log p(x₀), almost always replaced in practice by the unweighted noise-prediction loss L_simple.

Forward Chain (Fixed)

q(x_t | x_t−1) = N(√(1−β_t) x_t−1, β_t I), with β_t linearly spaced in the original work. Because the chain is Gaussian,

q(x_t | x₀) = N(√ᾱ_t x₀, (1−ᾱ_t) I), x_t = √ᾱ_t x₀ + √(1−ᾱ_t) ε.

No learnable parameters live in q. That is why DDPM training is so stable compared to GANs: the inference path is not a second network.

Reverse Chain (Learned) and the ELBO

We want p_θ(x₀) = ∫ p(x_T) ∏_t=1..T p_θ(x_t−1 | x_t) dx_1:T with p(x_T) = N(0, I). Each reverse step is Gaussian:

p_θ(x_t−1 | x_t) = N(x_t−1; μ_θ(x_t, t), Σ_θ(x_t, t)).

Ho et al. fix Σ to a constant schedule (β_t I or the posterior β̃_t I) and parameterize μ_θ through ε_θ:

μ_θ(x_t, t) = (1/√α_t) ( x_t − (β_t / √(1−ᾱ_t)) ε_θ(x_t, t) ).

The variational bound then decomposes into KL terms between Gaussians (analytic). After dropping constants and reweighting, the working objective is:

L_simple = E_{t, x₀, ε} [ || ε − ε_θ(√ᾱ_t x₀ + √(1−ᾱ_t) ε, t) ||² ].

Algorithm 1 — Training

1. Draw x₀

From the dataset (or z₀ = scale·E(x)).

2. Draw t, ε

t uniform in 1..T; ε ~ N(0,I).

3. Form x_t

Closed-form q_sample.

4. Gradient step

∇_θ ||ε − ε_θ(x_t, t)||².

import torch
import torch.nn.functional as F

def ddpm_train_step(model, x0, alphas_cumprod, optimizer):
    """One minibatch of L_simple. model(x_t, t) -> eps_hat."""
    N = x0.size(0)
    T = alphas_cumprod.numel()
    t = torch.randint(0, T, (N,), device=x0.device)
    eps = torch.randn_like(x0)
    a_bar = alphas_cumprod[t].view(N, *([1] * (x0.ndim - 1)))
    xt = a_bar.sqrt() * x0 + (1.0 - a_bar).sqrt() * eps
    eps_hat = model(xt, t)
    loss = F.mse_loss(eps_hat, eps)
    optimizer.zero_grad(set_to_none=True)
    loss.backward()
    optimizer.step()
    return float(loss)

# In SD-class training, replace x0 with scaled VAE latents and pass text cond into model.

Algorithm 2 — Ancestral Sampling

Start from x_T ~ N(0, I). For t = T, …, 1:

x_t−1 = (1/√α_t) (x_t − (β_t/√(1−ᾱ_t)) ε_θ(x_t,t)) + σ_t z, z ~ N(0,I) (z=0 if t=1).

Ho et al. use σ_t = √β_t or the true posterior std √β̃_t. This extra z is why DDPM samples are stochastic even given ε_θ and x_T. T = 1000 network calls per image is the historical pain point—DDIM attacks it without retraining.

@torch.no_grad()
def ddpm_sample(model, shape, alphas, alphas_cumprod, device="cuda"):
    T = alphas.numel()
    x = torch.randn(shape, device=device)
    for t in range(T - 1, -1, -1):
        eps = model(x, torch.full((shape[0],), t, device=device, dtype=torch.long))
        alpha_t = alphas[t]
        a_bar_t = alphas_cumprod[t]
        beta_t = 1.0 - alpha_t
        mean = (1.0 / alpha_t.sqrt()) * (x - (beta_t / (1.0 - a_bar_t).sqrt()) * eps)
        if t == 0:
            x = mean
        else:
            x = mean + beta_t.sqrt() * torch.randn_like(x)
    return x

Original Paper vs Production SD

Piece	Ho et al. 2020	SD 1.5-class production
Space	Pixels (e.g. 32×32, 256×256)	Scaled VAE latent 4×64×64
T / β	1000, linear 1e-4→0.02	Same discrete math; cosine / scaled-linear variants exist
Network	UNet, GN, self-attn at 16×16	UNet + cross-attention to CLIP
Loss	L_simple on ε	Same, plus cond dropout for CFG
Sample steps	1000 ancestral	Often 20–50 DDIM/DPM++ in the UI

Why DDPM stuck

MSE training is boring—in a good way.
Mode coverage beats typical GANs.
Conditioning slots in cleanly (class, text, mask).

Why UIs abandoned pure ancestral DDPM

1000 steps is too slow for interactive stills.
Stochastic z hurts exact reproducibility.
Better ODE solvers reuse the same ε_θ.

Common Misconception

“L_simple is the ELBO, and DDPM sampling must always use T=1000.” L_simple is a reweighted, simplified bound—it drops the theoretically correct per-timestep weights and works better in practice. Sampling step count is a discretization choice: you can evaluate a DDPM-trained net with 50 DDIM steps. Conversely, calling a 50-step UI sampler “DDPM” is sloppy; ancestral DDPM implies the stochastic Markov reverse with σ_t z.

Knowledge Check

Short Answer: Are there learnable parameters in the DDPM forward process q? Answer: No—q is a fixed Gaussian Markov chain.
True/False: L_simple is MSE between true ε and ε_θ(x_t, t). Answer: True.
Multiple Choice: Original Ho et al. T and linear β range: (a) T=50, β 0.1→0.9, (b) T=1000, β 1e-4→0.02, (c) T=20, cosine only. Answer: (b).
Short Answer: Write μ_θ in terms of x_t, ε_θ, α_t, β_t, ᾱ_t. Answer: μ_θ = (1/√α_t)(x_t − (β_t/√(1−ᾱ_t)) ε_θ).
True/False: During training you must run the reverse chain to compute the loss. Answer: False—training uses one random t and q_sample.
Multiple Choice: The extra z in ancestral sampling makes DDPM: (a) deterministic, (b) stochastic, (c) a GAN. Answer: (b).
Short Answer: What is p(x_T)? Answer: Standard normal N(0, I) (isotropic Gaussian prior).
Short Answer: What changes when you train DDPM on SD latents instead of pixels? Answer: x₀ becomes scaled z₀ = E(x)·scaling_factor; the algorithms stay the same (plus text cond).
Multiple Choice: The ELBO terms between Gaussians are: (a) intractable Monte Carlo only, (b) analytic KLs, (c) adversarial. Answer: (b).
True/False: A checkpoint trained with L_simple can only be sampled with ancestral DDPM. Answer: False—DDIM and other solvers reuse ε_θ.

Key Takeaways

DDPM = fixed Gaussian forward chain + learned Gaussian reverse + L_simple on noise.
Train with random t and closed-form x_t; sample by walking t downward with optional extra noise z.
Original recipe: T=1000, linear β, UNet. SD keeps the math and moves it to latents + text.
L_simple is a practical reweighting of the ELBO, not a different generative model.
Next: DDIM—same ε_θ, fewer steps, deterministic option.

Trainer’s Guide

Hands-on idea: Train a 20-minute MNIST DDPM (tiny UNet, T=200). Compare 200-step ancestral samples vs the same net at 20 ancestral steps (quality collapse) vs previewing DDIM at 20 steps next lecture.

Discussion prompt: If L_simple ignores the theoretically correct ELBO weights, why did Ho et al. still prefer it? (Hint: it up-weights harder mid/high-noise terms that matter for sample quality.)

Recap: DDPM turns diffusion into MSE training and stochastic ancestral sampling. Continue with DDIM.