Video — Module 16.1

How This Lesson Fits the Module & Volume

Vision handled still frames. Video is vision plus time: motion, shot changes, speech tracks, and long context. This lecture splits the modality into understanding (what happened) vs generation (synthesize clips). Deep dives: video understanding, video generation, lip sync. Product generators (Sora, Runway, Pika, Veo, …) live in Module 16.4.

Video almost always bundles other 16.1 modalities: frames (vision), soundtrack (speech / audio), sometimes burned-in text (OCR).

Learning Objectives

By the end of this lesson, students should be able to:

Define video as a spatiotemporal modality, not a bag of independent JPEGs.
Separate video understanding from video generation and lip-sync.
Explain sampling, keyframes, and why naive frame-by-frame fails.
Sketch a multimodal video pipeline (frames + STT + optional OCR).
Know that 16.4 catalogs generators; 16.1 defines the task.
Connect ViT/CNN frame encoders (Vol. 07 / Vol. 10) to temporal models.

Definition

Video is a sequence of frames plus optional audio, indexed by time. Video understanding maps that sequence to labels, captions, events, or retrieval. Video generation maps text, images, or motion controls to new frames. Both are capabilities; they share encoders but not losses, latency, or safety profiles.

Why Time Changes Everything

Still vision	Video extra problem	Typical fix
One image context	Minutes of frames (token explosion)	Keyframe / clip sampling
No motion	Actions, causality, tracking	Temporal attention / 3D CNN / memory
No soundtrack	Speech, music, SFX	STT + audio tagging in parallel
Single layout	Shots, cuts, camera move	Shot detection before captioning

Two Families of Tasks

Understand

Action recognition, highlight detect
Video Q&A / chaptering
Moderation, search, sports analytics
Lecture: Video understanding

Generate

Text/image-to-video
Extend / restyle a clip
Lip-sync & talking avatars
Catalog: Sora, Runway, Pika

Align A/V

Lip sync
Dubbing + face motion
Avatar generation

Practical: Sample, Then Understand

Never dump every frame into a ViT. Sample, optionally run STT on the audio track, then ask a multimodal model.

# Sample 1 fps keyframes + optional STT, then caption the clip
import subprocess, tempfile
from pathlib import Path
from openai import OpenAI

client = OpenAI()

def extract_keyframes(video: str, fps: float = 1.0) -> list[Path]:
    out_dir = Path(tempfile.mkdtemp())
    subprocess.check_call([
        "ffmpeg", "-i", video, "-vf", f"fps={fps}",
        str(out_dir / "f_%04d.jpg"), "-hide_banner", "-loglevel", "error",
    ])
    return sorted(out_dir.glob("*.jpg"))

def understand_clip(video: str, question: str) -> str:
    frames = extract_keyframes(video, fps=0.5)[:8]  # cap tokens
    content = [{"type": "text", "text": question}]
    for p in frames:
        import base64
        b64 = base64.b64encode(p.read_bytes()).decode()
        content.append({
            "type": "image_url",
            "image_url": {"url": f"data:image/jpeg;base64,{b64}"},
        })
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": content}],
        max_tokens=500,
    )
    return resp.choices[0].message.content
# Pair with speech-to-text on the audio track for interviews / lectures.

Understand vs Generate — Ops Contrast

Understanding

Latency: seconds–minutes OK for batch
Eval: accuracy, recall@k, human QA
Safety: PII in frames + speech

Generation

Latency: expensive GPUs / queued jobs
Eval: fidelity, motion consistency, prompt adherence
Safety: deepfakes, likeness, copyright

Common Misconception

“Run CLIP on every frame and average.” Mean-pooled CLIP loses verbs, order, and who-did-what. Video understanding needs temporal structure (or at least shot-aware captions + STT). CLIP is still useful for keyframe retrieval—not a full video model.

Knowledge Check

Short Answer: What does video add on top of still vision? Answer: Time / motion / temporal context (and usually audio).
True/False: Video generation and video understanding use the same evaluation metrics. Answer: False—accuracy vs fidelity/motion/prompt adherence.
Multiple Choice: Sora and Runway belong in: (a) 16.1 capability map, (b) 16.4 product catalog, (c) Vol. 05 clustering. Answer: (b) (also named in 16.1 as examples).
Short Answer: Why sample keyframes instead of every frame? Answer: Token/compute explosion; redundant near-duplicate frames.
True/False: Lip sync is a form of A/V alignment, not full world-model video generation. Answer: True.
Multiple Choice: Interview video is best understood with: (a) frames only, (b) frames + STT, (c) TTS only. Answer: (b).
Short Answer: Which 16.1 lecture covers synthesizing new clips? Answer: Video generation.
True/False: Averaging CLIP embeddings across frames captures action order. Answer: False.
Multiple Choice: Burned-in subtitles in a video are primarily an: (a) OCR problem, (b) TTS problem, (c) k-NN problem. Answer: (a).
Short Answer: Where do you go for the STT catalog used on video soundtracks? Answer: Module 16.2.

Key Takeaways

Video = spatiotemporal modality; sample time deliberately.
Split understand vs generate vs A/V align (lip sync / avatars).
Combine vision + speech (+ OCR) rather than one magic endpoint.
16.4 lists generators; this lecture names the capability.
Next: Audio (including non-speech sound).

Trainer’s Guide

Lab: 60-second clip. Students must produce (1) 0.5 fps keyframe storyboard, (2) STT transcript, (3) a 5-sentence understanding summary. Compare to “upload whole file to a video LLM” and discuss cost.

Debate: When is 16.4 generation the right product vs editing real footage + lip sync?

Recap: Video adds time to vision and almost always pulls in speech. Continue with Audio.