Video Generation — Module 16.1

How This Lesson Fits the Module & Volume

Image generation makes one frame. Video generation makes a coherent sequence—motion, identity, lighting, and (often) implied physics—from text, an image, or a driving video. This is the capability lecture; the product catalog is Module 16.4 (Sora, Runway Gen-3, Pika, Kling, Veo, Luma, SVD, Hailuo).

Do not confuse generation with understanding, or with narrower A/V alignment (lip sync, avatars).

Learning Objectives

By the end of this lesson, students should be able to:

Define video generation and its main conditioning modes.
Contrast world-synthesis vs lip-sync vs avatar puppeteering.
Explain temporal consistency, identity drift, and duration limits.
Sketch a job-based API (submit → poll → download) in Python.
List safety/IP issues unique to moving likenesses.
Point to 16.4 for vendor comparison and Vol. 17 for generative theory.

Definition

Video generation synthesizes a frame sequence (and sometimes audio) from conditioning: text-to-video, image-to-video, video-to-video, or motion-controlled generation. Success means prompt adherence plus temporal coherence—objects should not morph or teleport without cause.

Conditioning Modes

Mode	Input	Strength / risk
Text → video	Prompt	Flexible; weakest control
Image → video	Still + prompt	Locks look; motion may invent
Video → video	Clip + prompt/style	Restyle / extend; identity drift
Camera / motion control	Trajectories, depth	Production control
Audio-driven (narrow)	Speech/song	Often lip-sync, not full worlds

Capability vs Adjacent Tasks

Full generation

New scenes / physics-ish motion
16.4 models (Sora, Veo, Runway…)
High compute, queued jobs

Lip sync

Existing face + new audio
Lip-sync lecture
Dubbing, localization

Avatar

Persistent digital presenter
Avatar lecture
Often TTS + lip sync stack

Practical: Async Job Pattern

Unlike image APIs, video jobs are almost always asynchronous. Teach the capability as submit/poll—vendor URLs change in 16.4.

# Conceptual async text-to-video job (pseudo-client; swap in 16.4 SDKs)
import time
from pathlib import Path

def generate_video(prompt: str, client, poll_s: int = 5, timeout_s: int = 600) -> Path:
    job = client.videos.create(prompt=prompt, duration_s=4, resolution="720p")
    job_id = job.id
    deadline = time.time() + timeout_s
    while time.time() < deadline:
        status = client.videos.get(job_id)
        if status.state == "succeeded":
            data = client.videos.download(job_id)
            out = Path(f"{job_id}.mp4")
            out.write_bytes(data)
            return out
        if status.state == "failed":
            raise RuntimeError(status.error)
        time.sleep(poll_s)
    raise TimeoutError(f"video job {job_id} timed out")

# Always store prompt, seed, model version, and safety flags with the mp4.

Quality Axes Unique to Video

Wanted

Temporal consistency (identity, lighting)
Believable motion / camera
Prompt + physics plausibility

Failure modes

Morphing limbs, extra fingers over time
Flicker, shot jumps, duration caps
Unconsented likeness / deepfake risk

Common Misconception

“Video generation is just image generation in a for-loop.” Independent txt2img frames do not share identity or motion. Temporal models (or image-to-video with explicit motion) are required. Likewise, a lip-sync tool is not Sora: it does not invent a city; it animates a mouth.

Knowledge Check

Short Answer: What extra requirement does video generation add beyond image gen? Answer: Temporal coherence / consistent motion and identity across frames.
True/False: Sora vs Runway vs Pika comparisons belong in 16.1, not 16.4. Answer: False—16.4 is the product catalog.
Multiple Choice: Image-to-video typically: (a) freezes the look and invents motion, (b) is OCR, (c) is STT. Answer: (a).
Short Answer: Why are video APIs usually async jobs? Answer: High compute / long runtime vs interactive image calls.
True/False: Lip sync is the same capability as text-to-video world synthesis. Answer: False.
Multiple Choice: Unconsented celebrity video is primarily a: (a) safety/likeness issue, (b) pooling layer, (c) BLEU score. Answer: (a).
Short Answer: Name two 16.4 example products. Answer: Sora, Runway, Pika, Kling, Veo, Luma, SVD, Hailuo (any two).
True/False: Generating independent frames with SDXL and concatenating them yields coherent video. Answer: False in general—no shared temporal model.
Multiple Choice: Video understanding vs generation: (a) read vs synthesize clips, (b) both are TTS, (c) both are OCR. Answer: (a).
Short Answer: Which lecture follows video generation? Answer: Voice cloning.

Key Takeaways

Video generation synthesizes temporally coherent clips from text/image/video.
Narrower cousins: lip sync and avatars—do not collapse them.
Treat APIs as jobs; log prompt, model, and safety metadata.
Vendor bake-off: Module 16.4.
Next: Voice cloning.

Trainer’s Guide

Whiteboard: Three columns—understand (16.1 later lecture), generate (this + 16.4), align (lip sync/avatar). Drop one product logo into generate only.

Ethics case: Political deepfake vs clearly labeled synthetic ad. Students write a release checklist (consent, watermark, disclosure).

Recap: Video generation is spatiotemporal synthesis; 16.4 names the models. Continue with Voice cloning.