ControlNet

How This Lesson Fits the Module & Volume

Prompts alone do not pin pose, edges, or camera depth. After Stable Diffusion, SDXL, and FLUX, ControlNet is how you inject a spatial hint into the denoiser without retraining the whole base. Volume 16 named ControlNet as an ecosystem layer; here you learn the mechanism: locked copy + zero convolutions + a conditioning image (Canny, pose, depth, …).

Adapters that change style or subject without a pose map are next: LoRA (Diffusion) and DreamBooth. Pixel edits without a new pose are inpainting. Production graphs live in ComfyUI and Automatic1111.

Learning Objectives

By the end of this lesson, students should be able to:

Define ControlNet as a trainable copy of the denoiser encoder that conditions on a spatial map.
Choose Canny, OpenPose, or depth (and know when scribble/seg/normal apply).
Run a diffusers ControlNet pipeline: preprocessor → conditioning scale → prompt.
Explain zero-initialized convolutions and why the base checkpoint stays frozen at the start.
Contrast ControlNet (layout) with LoRA (style/subject) and inpaint (masked pixels).
Plan multi-ControlNet stacks and UI wiring in ComfyUI / A1111 without inventing FLOPs.

Definition

ControlNet (Zhang, Rao, Agrawala, 2023) clones the UNet encoder (and mid block) of a frozen diffusion backbone, processes a conditioning image (edges, pose skeleton, depth, normals, segmentation, scribble, …), and injects those features into the locked UNet through zero-initialized convolutions. At step 0 the clone adds nothing, so the pretrained generator is not destroyed; training then learns spatial control. FLUX and other DiTs use ControlNet-like or ControlNet ports—same job, different tensor shapes.

Why Prompts Are Not Enough

Text says “a person sitting.” It does not say which joint angles or where the horizon sits. ControlNet turns a cheap 2D hint into extra channels the denoiser must respect, while the prompt still owns appearance, lighting, and style. That split is why product teams storyboard with pose/depth first, then dress the scene with prompt + LoRA.

Conditioner	Input map	Typical use
Canny	Edge image (Canny detector)	Keep object outlines / logo silhouettes; restyle interiors
OpenPose	Body / hand / face keypoints	Character pose, dance, product-model stance
Depth	MiDaS / ZoeDepth / metric depth	Camera layout, furniture placement, relight-friendly geometry
Normal / scribble / seg	Surface normals, hand sketch, ADE20K-style labels	Material, rough ideate, semantic layout

Architecture Sketch

1. Freeze base

UNet (or DiT) weights locked

2. Clone encoder

Trainable copy + hint encoder

3. Zero convs

Add features without a shock at init

4. Denoise

Prompt + map + timestep

Canny ControlNet in diffusers

Always run the same preprocessor at train and infer. A pose ControlNet fed Canny edges is garbage. controlnet_conditioning_scale (often 0.5–1.2) is the knob between “suggestion” and “prison.”

from diffusers import StableDiffusionControlNetPipeline, ControlNetModel, UniPCMultistepScheduler
from diffusers.utils import load_image
import torch
import cv2
import numpy as np
from PIL import Image

controlnet = ControlNetModel.from_pretrained(
    "lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16
)
pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    controlnet=controlnet,
    torch_dtype=torch.float16,
)
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
pipe = pipe.to("cuda")

src = load_image("chair_photo.png").convert("RGB")
arr = np.array(src)
edges = cv2.Canny(arr, 100, 200)
canny = Image.fromarray(edges)

image = pipe(
    prompt="oak dining chair, soft studio light, no text",
    image=canny,
    num_inference_steps=20,
    controlnet_conditioning_scale=0.9,
).images[0]
image.save("canny_chair.png")

# Pose: OpenPose preprocessor + sd-controlnet-openpose
# Depth: depth map + sd-controlnet-depth
# SDXL: ControlNetModel + StableDiffusionXLControlNetPipeline

ControlNet vs LoRA vs Inpaint

Spatial layout / pose / edges
Needs a hint image
Base look mostly unchanged

LoRA

Style, character, medium
Tiny adapter weights
No pose map required

Inpaint

Replace masked pixels
Unmasked region stays
Edit, not full restage

Strengths and Tradeoffs

Strengths

Production control without full FT
Composable (multi-ControlNet + LoRA)
Huge SD 1.5 zoo; XL/FLUX catching up

Tradeoffs

Wrong preprocessor = silent failure
Scale too high → stiff, “traced” look
Extra VRAM vs prompt-only

Related Lectures

Lecture	Why it sits beside ControlNet
FLUX / SDXL	Backbones you attach a ControlNet (or port) to
LoRA (Diffusion)	Style/subject PEFT; often stacked with ControlNet
Inpainting	Pixel mask instead of a pose/edge map
ComfyUI / A1111	Where preprocessors + models are wired in prod
Vol. 16 SD catalog	Ecosystem naming without the zero-conv math

Common Misconception

“ControlNet fine-tunes the whole Stable Diffusion model.” The original design freezes the pretrained UNet and trains the clone + zero convs. You can later train or LoRA the base, but that is a different recipe. Second mistake: treating Canny, pose, and depth as interchangeable files—each checkpoint expects its own map family.

Knowledge Check

Short Answer: What spatial hints do Canny, OpenPose, and depth provide? Answer: Edges/outlines; body keypoints/pose; camera/scene geometry.
True/False: Classic ControlNet freezes the original UNet and trains a cloned encoder path. Answer: True.
Multiple Choice: Zero convolutions exist to: (a) tokenize audio, (b) inject control without wrecking the pretrained generator at init, (c) replace the VAE. Answer: (b).
Short Answer: Name the diffusers knob that sets how strongly the map is followed. Answer: controlnet_conditioning_scale (conditioning scale).
True/False: You should feed a depth ControlNet a Canny image. Answer: False—preprocessor must match the checkpoint.
Multiple Choice: ControlNet vs LoRA: ControlNet primarily changes: (a) spatial layout, (b) tokenizer BPE ranks, (c) Redis TTL. Answer: (a).
Short Answer: Why stack ControlNet with a style LoRA? Answer: Map owns pose/layout; LoRA owns look/subject.
True/False: FLUX has no possible ControlNet-like ports. Answer: False—community/BFL-style control ports exist; shapes differ from SD 1.5.
Multiple Choice: Best UI siblings for wiring ControlNet: (a) ComfyUI / A1111, (b) Excel Solver, (c) Vol. 09 CBOW. Answer: (a).
Short Answer: What comes next for low-rank style adapters? Answer: LoRA (Diffusion).

Key Takeaways

ControlNet adds spatial conditioning (Canny, pose, depth, …) via a cloned encoder + zero convs.
Match preprocessor to checkpoint; tune conditioning scale.
Layout ≠ style: pair with LoRA; pixel holes belong to inpaint.
SD 1.5 zoo is deepest; XL/FLUX control is real but not identical weights.
Continue with LoRA (Diffusion).

Trainer’s Guide

Lab: One photo → Canny restyle, OpenPose new costume, depth relight. Sweep conditioning scale 0.4 vs 1.2. Optionally load a style LoRA on the pose run.

Whiteboard: Draw frozen UNet, clone, zero conv, hint image. Label “prompt = appearance, map = geometry.” Arrow to LoRA and inpaint.

Recap: ControlNet pins pose, edges, and depth while the prompt still paints. Continue with LoRA (Diffusion).