Image Inpainting — Module 17.1

How This Lesson Fits the Module & Volume

Generation so far assumed a full canvas: SD, SDXL, FLUX, dressed with ControlNet, LoRA, or DreamBooth. Inpainting is surgical: keep most of an image, regenerate only a mask. It reuses the same denoising story in latent space—noise the hole, condition on the unmasked context.

Outpainting is the sibling that expands the canvas (mask = new border). Both ship in ComfyUI and Automatic1111. Do not DreamBooth a whole identity just to swap a logo.

Learning Objectives

By the end of this lesson, students should be able to:

Define inpainting as masked denoising: regenerate mask, preserve context.
Describe how SD inpaint UNets take extra channels (masked latent + mask).
Run StableDiffusionInpaintPipeline (and know XL/FLUX inpaint cousins exist).
Tune strength / steps / mask dilation without destroying seams.
Contrast inpaint vs txt2img, img2img, ControlNet, and DreamBooth.
Prepare the next lecture: outpaint = inpaint on a padded canvas.

Definition

Image inpainting (in diffusion) fills a user-specified mask by denoising that region while conditioning on the unmasked pixels and a text prompt. Dedicated SD inpaint checkpoints widen the UNet’s first conv to accept concatenated noisy latent + masked-image latent + downsampled mask (9 channels vs 4). Generic img2img can fake inpaint by noising the whole image and hoping the unmasked area returns—specialized inpaint models + mask blending are why production editors look clean.

Mask + Denoise

1. Image + mask

White = edit, black = keep

2. Encode

VAE latent of image (and masked image)

3. Noise the hole

Strength sets how far from data

4. Denoise

Prompt + mask channels

5. Composite

Optional pixel blend on seams

Mode	What changes	What stays
txt2img	Entire canvas from noise	Nothing from a source photo
img2img	Whole image, strength-controlled	Composition roughly, not pixel-exact
Inpaint	Masked region only (ideally)	Unmasked pixels / structure
Outpaint	New border pixels	Original rectangle (approximately)

SD inpaint pipeline

runwayml/stable-diffusion-inpainting (1.5 family) and SDXL inpaint checkpoints are the usual HF ids—confirm the card. Mask convention in diffusers: white = inpaint, black = keep (always verify; some UIs invert). strength near 1.0 fully regenerates the hole; low strength barely moves it. Dilate the mask a few pixels so seams do not halo.

from diffusers import StableDiffusionInpaintPipeline
from diffusers.utils import load_image
import torch
from PIL import Image

pipe = StableDiffusionInpaintPipeline.from_pretrained(
    "runwayml/stable-diffusion-inpainting",
    torch_dtype=torch.float16,
).to("cuda")

init = load_image("storefront.png").resize((512, 512))
mask = load_image("storefront_mask.png").resize((512, 512)).convert("L")
# white pixels in mask = region to replace (e.g. old signage)

result = pipe(
    prompt="hand-painted wooden shop sign reading HARBOR, daylight, no extra logos",
    negative_prompt="watermark, unreadable letters, extra hands",
    image=init,
    mask_image=mask,
    num_inference_steps=30,
    guidance_scale=7.5,
    strength=0.92,
).images[0]
result.save("storefront_inpaint.png")

# SDXL: StableDiffusionXLInpaintPipeline + an XL inpaint checkpoint
# FLUX: community/BFL inpaint or fill workflows—same mask idea, different tensors

Inpaint vs the Rest of the Toolkit

Inpaint

Local pixel surgery
Prompt describes the hole
Fast vs new DreamBooth

ControlNet

Restage pose/edges globally
Hint map, not a keep-mask
Can guide inpaint too

DreamBooth / LoRA

New identity or style
Not a one-hole edit
Use if the subject must recur

Seams, Strength, and Mask Craft

Practices that help

Feather or dilate mask; avoid 1-px jagged holes
Prompt the fill, not the whole original scene
Match resolution to the inpaint checkpoint native size
Optional second pass at lower strength to blend

Failure modes

Inverted mask → you destroyed the keep region
Strength too low → ghost of the old object
Wrong checkpoint (txt2img UNet as inpaint)
Color/lighting mismatch at the boundary

Related Lectures

Lecture	Why it sits beside inpainting
Outpainting	Same pipeline; mask is the new border
Denoising / Latent space	Noise + VAE mechanics inside the hole
DreamBooth	Heavier alternative when identity must persist
ControlNet	Optional edge/depth guide inside the mask
ComfyUI / A1111	Brush masks and inpaint tabs/nodes

Common Misconception

“Any img2img run is inpainting.” Img2img noising the full frame will drift unmasked pixels. Real inpaint models see a mask channel (or you composite after a masked denoise). Second mistake: using a txt2img UNet and wondering why the hole ignores the mask. Third: painting a tiny mask then prompting the entire original caption—describe what should appear in the hole.

Knowledge Check

Short Answer: In diffusion inpainting, what do the mask and the prompt each control? Answer: Mask = which pixels to regenerate; prompt = what should fill that region.
True/False: SD inpaint UNets often take extra channels (masked latent + mask) beyond 4 latent channels. Answer: True.
Multiple Choice: diffusers class for 1.5 inpaint: (a) StableDiffusionInpaintPipeline, (b) PCA, (c) WhisperProcessor. Answer: (a).
Short Answer: Typical diffusers mask color for “edit this”? Answer: White (black = keep)—always verify the UI/pipeline.
True/False: strength near 1.0 more fully regenerates the masked region. Answer: True.
Multiple Choice: Outpainting is closest to: (a) inpaint on a padded canvas, (b) k-means, (c) MCP resources. Answer: (a).
Short Answer: Why dilate/feather a mask? Answer: Reduce halo/seams where new pixels meet old ones.
True/False: DreamBooth is the default tool to change one sign on a storefront. Answer: False—inpaint is the local edit.
Multiple Choice: txt2img vs inpaint: txt2img starts from: (a) full noise, no source keep-mask, (b) only the EXIF, (c) a frozen LLM. Answer: (a).
Short Answer: Name the next sibling lecture. Answer: Image Outpainting.

Key Takeaways

Inpaint = mask + denoise; unmasked context should stay.
Use an inpaint checkpoint / pipeline, not a naive full img2img.
Strength, mask craft, and fill prompt decide seam quality.
Cheaper than DreamBooth for one-off pixel surgery.
Continue with Image Outpainting.

Trainer’s Guide

Lab: Mask a coffee-cup logo; inpaint a new mark at strength 0.6 vs 0.95. Invert the mask once on purpose and discuss the disaster. Optional: SDXL inpaint vs 1.5 on the same photo.

Whiteboard: 4-ch txt2img UNet vs 9-ch inpaint UNet. Draw composite step. Arrow to outpaint as “mask the new border.”

Recap: Inpainting regenerates a mask while the rest of the photo holds. Continue with Image Outpainting.