Image Generation — Module 16.1

How This Lesson Fits the Module & Volume

So far 16.1 mostly read modalities. Image generation is the capability of synthesizing new pixels from text, images, or both (txt2img, img2img, inpaint). Architectures (GANs, diffusion) deepen in Volume 17. This lecture names the task, controls, and eval—not the vendor shelf. Products—DALL·E, Midjourney, Stable Diffusion, SDXL, Flux, Imagen, Firefly—are Module 16.3.

Generation still uses vision backbones: U-Nets/DiTs borrow CNN and ViT ideas; many systems condition on CLIP text embeddings (Vol. 07 CLIP + Vol. 09 embedding geometry).

Learning Objectives

By the end of this lesson, students should be able to:

Define image generation vs understanding vs editing.
Contrast txt2img, img2img, inpaint, outpaint, and upscale.
Explain conditioning (prompts, CLIP, ControlNet-style guides) at a capability level.
Call a generation API and save artifacts with prompt metadata.
List eval axes: fidelity, prompt adherence, safety, IP, brand.
Defer product bake-offs to Module 16.3 and theory to Vol. 17.

Definition

Image generation produces new pixel arrays from a conditioning signal (text, image, sketch, depth, style). It is the inverse of recognition: understanding maps pixels→labels; generation maps labels/conditions→pixels.

Task Variants

Variant	Input	Typical use
Text-to-image	Prompt (± negative)	Concept art, ads, mockups
Image-to-image	Image + prompt + strength	Restyle, mild edit
Inpaint / outpaint	Mask + prompt	Remove object, extend canvas
Upscale / restore	Low-res image	Print, archive
Conditioned (pose/depth/canny)	Guide map + prompt	Layout control, product shots

Conditioning Stack (Capability View)

Text encoder

CLIP / T5 / LLM prompts
Vol. 09 embeddings + Vol. 07 CLIP
Prompt hygiene beats magic words

Generator

Diffusion / DiT / GAN / AR
Vol. 17 for the math
16.3 for which product

Guides & safety

Control maps, IP-adapters
NSFW / likeness filters
Provenance / C2PA where required

Practical API Pattern

# Capability-level txt2img (OpenAI Images API; swap vendor in 16.3)
import json
from pathlib import Path
from openai import OpenAI

client = OpenAI()

def generate_image(prompt: str, out_dir: str = "gen_out") -> dict:
    Path(out_dir).mkdir(exist_ok=True)
    result = client.images.generate(
        model="gpt-image-1",
        prompt=prompt,
        size="1024x1024",
        n=1,
    )
    # Persist prompt + model id with the file (audit / reproduce)
    meta = {"prompt": prompt, "model": "gpt-image-1", "size": "1024x1024"}
    img_b64 = result.data[0].b64_json
    import base64
    png = Path(out_dir) / "image.png"
    png.write_bytes(base64.b64decode(img_b64))
    (Path(out_dir) / "meta.json").write_text(json.dumps(meta, indent=2))
    return {"path": str(png), "meta": meta}

# Agent rule: never claim the photo is a real customer; label as generated.

Eval and Governance

Optimize for

Prompt adherence + visual quality
Brand consistency / control
Reproducible seeds + logged prompts

Do not ignore

Likeness / deepfake policy
Copyrighted characters and styles
Confusing generation with “a photo of reality”

Common Misconception

“Image generation = Midjourney.” Midjourney is one 16.3 product. The capability also includes SDXL, DALL·E, Firefly, Flux, and on-prem diffusion. Conversely, running CLIP similarity is not generation—that is retrieval/understanding.

Knowledge Check

Short Answer: How does generation invert recognition? Answer: Recognition maps pixels→labels; generation maps conditions→pixels.
True/False: Module 16.1 is where you memorize every 16.3 vendor feature. Answer: False—16.1 is the capability; 16.3 is the catalog.
Multiple Choice: Inpainting requires: (a) a mask + prompt, (b) only TTS, (c) k-means. Answer: (a).
Short Answer: Which volume covers diffusion math in depth? Answer: Volume 17.
True/False: CLIP text embeddings are often used to condition generators. Answer: True.
Multiple Choice: Logging prompts with outputs is: (a) optional vanity, (b) needed for audit/repro, (c) OCR. Answer: (b).
Short Answer: Name two image-gen variants besides txt2img. Answer: Img2img, inpaint, outpaint, upscale, ControlNet-style (any two).
True/False: A generated product photo is automatically documentary evidence. Answer: False—label synthetic media.
Multiple Choice: SDXL and DALL·E lectures live in: (a) 16.2, (b) 16.3, (c) 15.4. Answer: (b).
Short Answer: Next lecture after image generation? Answer: Video generation.

Key Takeaways

Image generation synthesizes pixels from prompts and guides.
Variants: txt2img, img2img, inpaint, control, upscale.
CLIP/ViT/CNN skills still matter; Vol. 17 covers diffusion.
Compare vendors in 16.3, not here.
Next: Video generation.

Trainer’s Guide

Lab: Same brief, three capabilities: (1) retrieve a stock photo via CLIP, (2) generate txt2img, (3) inpaint a logo. Students write when each is legal/appropriate.

Preview 16.3: One slide listing DALL·E / MJ / SDXL / Flux—no deep product dive yet.

Recap: Image generation is pixels-out vision. Continue with Video generation.