Vision — Module 16.1

How This Lesson Fits the Module & Volume

After speech, agents need a second sense: vision—pixels that encode scenes, documents, UI, and people. You already built the machinery in Vol. 07 CNNs, Vol. 10 ViT, and Vol. 07 CLIP (with text-side intuition from Vol. 09 sentence embeddings). Here we treat vision as a modality and a family of capabilities, not a new architecture course.

Module 16.1 will drill OCR, captioning, image generation, ViT, and CLIP as capabilities. Product image models (DALL·E, SDXL, Midjourney, …) wait in Module 16.3. Video is next as a temporal extension of vision.

Learning Objectives

By the end of this lesson, students should be able to:

Define vision as a pixel modality and list its core capabilities.
Map CNNs, ViTs, and CLIP to understand vs generate vs retrieve.
Choose among classification, detection, OCR, captioning, and VQA for a product task.
Call a multimodal vision API from Python as an agent tool.
Contrast still-image vision with video and OCR.
Point ahead to 16.3 for vendor image generators, not confuse them with the capability.

Definition

Vision (computer vision / visual understanding) is any task whose primary input is still images—arrays of pixels—and whose output is labels, boxes, masks, text, embeddings, or new pixels. Generation (new images) is a vision capability with a different evaluation story than recognition.

Backbones You Already Know

Family	Inductive bias	Volume recap	16.1 use
CNN	Local filters, hierarchy	Vol. 07 CNN	Detection, OCR crops, mobile vision
ViT	Patches + self-attention	Vol. 10 ViT, 16.1 ViT	Modern image encoders
CLIP	Image–text contrastive space	Vol. 07 CLIP, Vol. 09 embeddings	Retrieval, zero-shot labels, 16.1 CLIP

Capability Menu

Understand

Classify / detect / segment
OCR (text in images)
Captioning & VQA

Align / retrieve

CLIP embeddings
Image–text search
Open-vocabulary labels

Generate

Image generation
Edit / inpaint / upscale
Catalog in 16.3

Vision as an Agent Tool

Same Vol. 15 pattern as speech: the LLM does not “see” unless you pass pixels or a caption/OCR dump into working memory. Multimodal chat models fuse vision internally; classic agents call a vision tool and store the text result.

# Vision tool: image -> structured observation for an agent
from openai import OpenAI

client = OpenAI()

def inspect_image(url: str, question: str) -> str:
    """VQA / caption-style observe step."""
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": question},
                {"type": "image_url", "image_url": {"url": url}},
            ],
        }],
        max_tokens=400,
    )
    return resp.choices[0].message.content

# Agent prompt example:
# "You cannot see. Call inspect_image(url, question) then answer from the returned text."
# For documents, prefer OCR (next lectures) over a vague caption.

When Vision Is the Wrong Tool

Use vision when

The ground truth is in pixels (UI, photo, diagram)
You need boxes, masks, or visual search
A human would look, not read a PDF text layer

Do not use vision when

A digital text layer already exists (parse the PDF)
You only need a stock illustration (that is generation)
The question is temporal (that is video)

Common Misconception

“Vision = image generation.” Generation is one capability. Most production vision systems read images (QC, OCR, moderation, retrieval). DALL·E and SDXL belong to 16.3; classification and CLIP retrieval are still vision.

Knowledge Check

Short Answer: What is the primary input of the vision modality? Answer: Still images / pixel arrays.
True/False: Volume 16.1 retrains CNNs from scratch. Answer: False—it maps existing Vol. 07/10 models to capabilities.
Multiple Choice: Open-vocabulary image–text matching is primarily: (a) CLIP, (b) k-means, (c) TTS. Answer: (a).
Short Answer: Name three understand-side vision capabilities. Answer: Classification, detection, segmentation, OCR, captioning, VQA (any three).
True/False: If a PDF has a text layer, you should still OCR every page first. Answer: False—parse digital text; OCR scans/photos.
Multiple Choice: Vendor image generators are catalogued in: (a) 16.2, (b) 16.3, (c) 15.4. Answer: (b).
Short Answer: Which volume introduced CNNs? Answer: Volume 07.
True/False: Video is just “many independent images” with no extra problem. Answer: False—time, motion, and long context matter (next lecture).
Multiple Choice: ViT treats an image as: (a) a bag of words, (b) a sequence of patches, (c) a spectrogram. Answer: (b).
Short Answer: Why pass vision tool output into working memory? Answer: So the text agent can reason, log, and retrieve; pixels alone are not a durable record.

Key Takeaways

Vision = still-image modality: understand, align, or generate.
CNNs (Vol. 07), ViT (Vol. 10), CLIP (Vol. 07 + Vol. 09 embeddings) are the backbones.
Agent pattern: vision tool → text/structured observe → same Vol. 15 loop.
16.3 is the image product catalog; this lecture is the capability map.
Next: Video adds time.

Trainer’s Guide

Lab: Same helpdesk agent: user sends a screenshot. Students must choose caption vs OCR vs detection and justify. Ban “just send it to GPT” without naming the capability.

Whiteboard: Pixel pipeline: CNN/ViT encoder → task head (class / boxes / text / CLIP space / diffusion). Circle which modules (16.3 vs 16.1) own each box.

Recap: Vision is the still-image modality built on Vol. 07/09/10. Continue with Video.