Denoising — Module 17.1

How This Lesson Fits the Module & Volume

Noise destroys structure. Denoising is the learned inverse: a network that, given x_t and t (and often text), estimates how to take a step toward x₀. This is the heart of every Vol. 16 generator you ran—SD, SDXL, SVD—and of DDPM / DDIM samplers. Latent space only changes where you denoise (z instead of pixels), not the idea.

You already know U-shaped CNNs from Vol. 07 and attention from Vol. 10. The denoiser is those blocks plus a time embedding. CFG is a sampling-time trick on top, not a different network family.

Learning Objectives

By the end of this lesson, students should be able to:

State the reverse-step goal: parameterize p_θ(x_t−1 | x_t) using a neural net.
Compare ε-prediction, x₀-prediction, v-prediction, and score prediction.
Describe a UNet denoiser: encoder–decoder skips, time embedding, optional cross-attention.
Implement one reverse update from a predicted ε (DDPM-style) in PyTorch.
Explain classifier-free guidance as an extrapolation between cond and uncond ε.
Relate “steps” and “guidance” sliders in SD UIs to this lecture, not to training epochs.

Definition

Denoising in diffusion is estimating, from a noisy sample x_t (or latent z_t) and timestep t, a quantity that lets you construct a slightly cleaner x_t−1. The most common quantity is the noise ε_θ(x_t, t) itself. Equivalently you can predict the clean image x̂₀, the velocity v, or the Stein score ∇_x log p_t(x). A sampler is just a rule that turns that prediction into the next tensor and repeats until t = 0.

What the Network Is Allowed to Predict

Target	Intuition	Where you meet it
ε-prediction	Guess the Gaussian that was mixed into x₀	DDPM, SD 1.x default
x₀-prediction	Guess the clean image/latent directly	Some samplers; easy to visualize
v-prediction	Predict a velocity mixing x₀ and ε	SD 2.x, some XL/zero-SNR setups
Score ∇ log p_t	Point uphill on the noisy density	NCSN / score-SDE papers

These targets are algebraically convertible for a Gaussian forward process. If you know ε̂ and ᾱ_t, then x̂₀ = (x_t − √(1−ᾱ_t) ε̂) / √ᾱ_t. Samplers in diffusers expose prediction_type; mismatching it with the checkpoint is a silent quality killer.

The Denoiser Architecture

UNet backbone

Downsample → mid → upsample.
Skip connections (Vol. 07 spatial hierarchy).
Self-attention at coarser maps.

Time embedding

Sinusoidal t, like Transformer positions.
Added or FiLM-ed into residual blocks.
Tells the net “which SNR am I at?”

Conditioning

Cross-attention to CLIP/T5 tokens.
Concat extra maps (mask, pose) later.
DiT / FLUX: transformer instead of UNet.

One Reverse Step from Predicted Noise

DDPM’s ancestral step (you will derive the variances in the DDPM lecture) looks like this once ε_θ is known. The extra z ~ N(0,I) is why two DDPM runs with the same seed still differ if you change step count; DDIM can drop that stochasticity.

import torch
from torch import nn

class TinyDenoiser(nn.Module):
    """Didactic CNN stand-in. Real SD uses a UNet + time + cross-attn."""
    def __init__(self, channels=3):
        super().__init__()
        self.net = nn.Sequential(
            nn.Conv2d(channels + 1, 64, 3, padding=1), nn.SiLU(),
            nn.Conv2d(64, 64, 3, padding=1), nn.SiLU(),
            nn.Conv2d(64, channels, 3, padding=1),
        )

    def forward(self, x, t_norm):
        # t_norm: (N,) in [0, 1], broadcast as an extra channel
        t_map = t_norm[:, None, None, None].expand(-1, 1, x.size(2), x.size(3))
        return self.net(torch.cat([x, t_map], dim=1))

@torch.no_grad()
def ddpm_step(x, t, eps_pred, alphas, alphas_cumprod, add_noise=True):
    alpha_t = alphas[t]
    a_bar_t = alphas_cumprod[t]
    beta_t = 1.0 - alpha_t
    mean = (1.0 / alpha_t.sqrt()) * (x - beta_t / (1.0 - a_bar_t).sqrt() * eps_pred)
    if t == 0 or not add_noise:
        return mean
    sigma = beta_t.sqrt()
    return mean + sigma * torch.randn_like(x)

# toy loop (pixel space, unconditioned)
T = 1000
betas = torch.linspace(1e-4, 0.02, T)
alphas = 1.0 - betas
alphas_cumprod = torch.cumprod(alphas, dim=0)
model = TinyDenoiser()
x = torch.randn(1, 3, 32, 32)
for t in range(T - 1, -1, -1):
    t_norm = torch.tensor([t / T])
    eps = model(x, t_norm)
    x = ddpm_step(x, t, eps, alphas, alphas_cumprod)
# x is a crude sample; swap TinyDenoiser for a trained UNet in production

Classifier-Free Guidance (CFG)

Ho & Salimans dropped the extra classifier. During training, randomly replace the text condition with a null token (~10% of the time). At sample time run the denoiser twice and set

ε̂ = ε_uncond + s · (ε_cond − ε_uncond)

s = 1 is the conditional model; s > 1 extrapolates past it (typical SD 1.5: ~7–8; SDXL often lower; FLUX schnell often ~1). Too high s burns contrast, drops diversity, and can ignore the prior—the “CFG look.” Negative prompts are just a different cond embedding in the uncond (or a second cond) slot.

Training vs Sampling Denoising

Training

Sample x₀, t, ε; build x_t.
Loss ≈ ||ε − ε_θ(x_t, t, c)||².
One net eval per example (plus cond dropout).

Sampling

Start from x_T ~ N(0, I).
Many net evals (steps × CFG doubles).
Sampler choice (DDPM/DDIM/DPM) changes the path, not the weights.

Common Misconception

“The UNet outputs the finished image in one forward pass, like a GAN generator.” One forward pass outputs a noise (or x₀) estimate at that t. A finished image is dozens to thousands of passes. Also: more steps are not monotonically better once you leave the stochastic DDPM regime—a good 20-step DPM++ run can beat a sloppy 100-step run. Match prediction_type and scheduler to the checkpoint.

Knowledge Check

Short Answer: Name two valid prediction targets for a diffusion denoiser. Answer: Any two of ε, x₀, v, score.
True/False: CFG trains a separate classifier on noisy images. Answer: False—classifier-free guidance drops the classifier and uses cond vs null.
Multiple Choice: Time embeddings exist so the UNet: (a) knows the batch size, (b) knows the noise level / SNR, (c) replaces the VAE. Answer: (b).
Short Answer: Given ε̂ and ᾱ_t, how do you recover x̂₀? Answer: x̂₀ = (x_t − √(1−ᾱ_t) ε̂) / √ᾱ_t.
True/False: SD 1.5 and SD 2.x always share the same prediction type. Answer: False—2.x often uses v-prediction.
Multiple Choice: Raising CFG scale s typically: (a) always increases diversity, (b) pushes toward the prompt and can oversaturate, (c) shortens training. Answer: (b).
Short Answer: Why does CFG roughly double inference cost? Answer: You evaluate the denoiser twice per step (conditioned and unconditioned).
Short Answer: What do UNet skip connections carry? Answer: High-resolution spatial detail from the encoder to the decoder (same idea as segmentation UNets).
Multiple Choice: A “steps” slider changes: (a) dataset epochs, (b) reverse-chain / ODE evaluations, (c) VAE channel count. Answer: (b).
True/False: DDPM reverse steps are deterministic if you still add z ~ N(0,I) at each t > 0. Answer: False—that extra z makes them stochastic; DDIM η=0 is the deterministic cousin.

Key Takeaways

Denoising is iterative: predict ε (or an equivalent), take a scheduler step, repeat.
Prediction type is part of the checkpoint contract; do not mix ε and v blindly.
The denoiser is a time-conditioned UNet or transformer, optionally cross-attending to text.
CFG extrapolates between null and text conditions; it is a sampler knob with real failure modes.
Next: Latent Space—why we denoise z, not RGB, in Stable Diffusion.

Trainer’s Guide

Hands-on idea: Freeze a tiny trained MNIST DDPM (or a diffusers pipeline on CPU with 5 steps). Visualize x_t and x̂₀ every few reverse steps. Students should see x̂₀ sharpen long before x_t looks clean.

Discussion prompt: At s = 1 vs s = 15, which images look more like “the training prior” and which look like “the prompt police”? When would you want each?

Recap: Denoising is a time-conditioned network plus a sampler rule—CFG steers it. Continue with Latent Space.