Video Understanding — Module 16.1

How This Lesson Fits the Module & Volume

This is the Module 16.1 capstone. Video introduced the modality; video understanding is the capability of mapping clips to meaning: actions, chapters, Q&A, search, moderation. It composes vision (keyframes / ViT), STT, optional OCR and audio tags, and language—exactly the agent observe path from Vol. 15, now multimodal.

It is not video generation (16.4 catalog). After this lecture you enter Module 16.2 Whisper and the STT product shelf.

Learning Objectives

By the end of this lesson, students should be able to:

Define video understanding vs generation vs lip-sync.
Design a multi-track pipeline: sample frames + STT + optional OCR/audio tags.
Choose temporal granularity (shot, scene, whole video).
Implement a practical understand-clip function in Python.
Eval with grounded Q&A / retrieval, not only a single caption.
Hand off to 16.2 for soundtrack STT products and to 16.4 only if synthesizing video.

Definition

Video understanding infers semantic structure from a spatiotemporal signal: action labels, event timelines, summaries, video Q&A, highlight detection, or embeddings for search. The output is language or labels about existing footage, not new pixels.

16.1 Capability Recap (Use Them Together)

Track	Capability	Lecture
Frames	ViT / CLIP / caption / detect	ViT, CLIP, captioning
Speech	STT + diarization	STT → 16.2
Non-speech sound	Audio tags	Audio
On-screen text	OCR	OCR
Time	Shots, memory, sampling	Video + this lecture

Task Menu

Label & retrieve

Action recognition
CLIP-on-keyframes search
Moderation / sports events

Describe & QA

Chaptering / summaries
Video Q&A grounded in time
Lecture/meeting minutes

Not this lecture

Sora / Runway / Pika → 16.4
Lip sync / avatars
Pure TTS without video

Practical Multi-Track Understand

# Compose 16.1 capabilities: STT + sampled captions -> grounded summary
from openai import OpenAI

client = OpenAI()

def understand_video(video_path: str, question: str,
                     transcribe_fn, keyframe_caption_fn) -> str:
    """transcribe_fn / keyframe_caption_fn are injected (Whisper in 16.2, VLM captions)."""
    transcript = transcribe_fn(video_path)          # speech track
    storyboard = keyframe_caption_fn(video_path, fps=0.5, max_frames=12)
    prompt = (
        "Answer ONLY from the transcript and storyboard. Cite timestamps if present.\n"
        f"QUESTION: {question}\n\nTRANSCRIPT:\n{transcript}\n\nSTORYBOARD:\n{storyboard}"
    )
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.1,
        max_tokens=500,
    )
    return resp.choices[0].message.content
# Add OCR(storyboard frames) if slides/credits matter. Add audio tags for alarms/music.
# Do not claim you "watched every frame" if you sampled at 0.5 fps — log the sample rate.

Eval and Ops

Do

Ground answers in transcript + frame captions
Log fps, models, and time ranges
PII-redact speech and faces per policy

Don’t

Mean-pool CLIP and call it understanding
Ignore the soundtrack on talk videos
Confuse this with 16.4 generation

Common Misconception

“A video LLM replaces STT, OCR, and sampling design.” Even when you use a native video model, you still need a system of record: transcript for search, keyframes for audit, sample rate for cost. Module 16.1 taught you to name those capabilities. 16.2–16.4 only swap implementations.

Knowledge Check

Short Answer: How does video understanding differ from video generation? Answer: Understanding describes existing footage; generation synthesizes new frames.
True/False: This lecture is the 16.4 product catalog. Answer: False—16.4 is generators; this is the understand capability.
Multiple Choice: Talk-show video should include: (a) STT + frames, (b) TTS only, (c) k-means only. Answer: (a).
Short Answer: Why log keyframe fps? Answer: Cost/audit; you did not actually see every frame.
True/False: Averaging CLIP embeddings captures action order. Answer: False.
Multiple Choice: Burned-in slide text is primarily: (a) OCR, (b) voice cloning, (c) vocoding. Answer: (a).
Short Answer: Which module starts immediately after 16.1 for STT products? Answer: Module 16.2 (Whisper first).
True/False: Lip sync is a video-understanding metric. Answer: False—it is A/V alignment / generation-adjacent.
Multiple Choice: Vol. 15 agents should store: (a) grounded transcript+storyboard recaps, (b) raw 4K only, (c) unconsented clones. Answer: (a).
Short Answer: Name two 16.1 tracks you would fuse for a factory CCTV clip with alarms and a supervisor speaking. Answer: Audio event tags + STT (+ optional frames/vision).

Key Takeaways

Video understanding = multimodal observe over time, not new video.
Fuse ViT/CLIP/captions + STT + OCR/audio tags; sample deliberately.
Ground language outputs; log what you actually computed.
16.2 next for STT vendors; 16.3 images; 16.4 generators.
Module 16.1 complete: modalities & capabilities named.

Trainer’s Guide

Capstone lab: 2-minute mixed clip (speech + on-screen text + a non-speech event). Students must run STT, 0.5 fps captions, OCR on text-y frames, audio tags, then answer three grounded questions. No 16.4 generation allowed.

Handoff: Open the 16.2 Whisper lecture as “now we pick an STT implementation for the soundtrack track you just used.”

Recap: Video understanding closes 16.1 by composing every modality into one observe pipeline. Continue with 16.2 Whisper (OpenAI).