Image Captioning — Module 16.1

How This Lesson Fits the Module & Volume

CLIP scores image–text pairs. Image captioning generates a natural-language description from pixels—the vision analog of STT. It uses a visual encoder (CNN/ViT, Vol. 07 / Vol. 10) plus a text decoder (Vol. 11 LM skills). Agents use captions as observe-text when they do not need OCR-exact strings. 16.3 is still generation of images, not captions.

Learning Objectives

By the end of this lesson, students should be able to:

Define captioning vs CLIP ranking vs OCR vs VQA.
Describe encoder–decoder (and VLM) caption architectures.
Choose captioning vs OCR for screenshots and documents.
Generate captions via a Python VLM/API call.
List eval metrics (BLEU/CIDEr/human) and hallucination risk.
Feed captions into Vol. 14 RAG / Vol. 15 agents as text observations.

Definition

Image captioning maps an image to a fluent sentence (or short paragraph) that describes salient content. Dense captioning labels regions; VQA answers a question; OCR transcribes writing. Captioning is generation in language, not generation of new pixels.

Task Boundaries

Task	Typical question	Must be exact?
Captioning	What is going on?	No—salient paraphrase
VQA	User-specified question	Depends
OCR	What characters appear?	Yes
CLIP	Which label/caption fits best?	Ranking, not decoding
Image gen (16.3)	Make pixels from text	Inverse direction

Architectures (Capability View)

Classic encoder–decoder

CNN/ViT features
LSTM/Transformer decoder
Show-Attend-Tell lineage

CLIP-guided

CLIP space + LM decoder
Good retrieval-then-caption
Still not OCR

VLM / multimodal LLM

ViT tokens into an LLM
Caption, VQA, UI agents
Watch hallucination

Practical Caption Tool

from openai import OpenAI

client = OpenAI()

CAPTION_PROMPT = (
    "Write one factual caption (max 40 words). "
    "Do not invent brands, numbers, or readable text. "
    "If there is writing, say 'contains text' and stop — caller should OCR."
)

def caption_image(url: str) -> str:
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": [
            {"type": "text", "text": CAPTION_PROMPT},
            {"type": "image_url", "image_url": {"url": url}},
        ]}],
        max_tokens=120,
        temperature=0.2,
    )
    return resp.choices[0].message.content.strip()

# Agent router: if user asks for totals / IDs -> OCR; if "describe this photo" -> caption.

Eval and Hallucination

Automatic metrics

BLEU / METEOR / CIDEr / SPICE
Useful for research baselines
Miss factual errors on rare objects

Production checks

Human spot-check + groundedness
Refuse to “read” tiny text
Store caption separately from OCR dump

Common Misconception

“A caption is as good as OCR for documents.” Captions summarize scenes; they will invent or skip numbers. Conversely, OCR of a landscape photo is useless. Route by question type. Also: CLIP similarity to a reference sentence is not captioning—it does not produce the sentence.

Knowledge Check

Short Answer: What does image captioning output? Answer: A natural-language description of the image.
True/False: CLIP zero-shot is the same as generating a caption. Answer: False—CLIP ranks; captioning decodes text.
Multiple Choice: Invoice totals should use: (a) OCR, (b) a poetic caption, (c) TTS. Answer: (a).
Short Answer: Which volumes supply encoder vs decoder skills? Answer: Vol. 07/10 vision encoder; Vol. 11-style language decoder / VLM.
True/False: Module 16.3 catalogs image captioning products. Answer: False—16.3 is image generation models.
Multiple Choice: VQA differs from captioning because: (a) it answers a specific question, (b) it clones voices, (c) it is k-means. Answer: (a).
Short Answer: Name one captioning hallucination risk. Answer: Invented brands, counts, readable text, objects not present (any one).
True/False: Captions can be chunked into Vol. 14 RAG like any text. Answer: True.
Multiple Choice: Dense captioning: (a) describes regions, (b) generates 16.4 video, (c) does STT. Answer: (a).
Short Answer: Last 16.1 lecture after captioning? Answer: Video understanding.

Key Takeaways

Captioning generates scene text from images; CLIP ranks; OCR transcribes.
Encoder (ViT/CNN) + decoder/VLM; low temperature for factual captions.
Never substitute captions for document strings.
Captions are agent observations—store and retrieve as text.
Next: Video understanding (captions over time + speech).

Trainer’s Guide

Lab: Three images—landscape, UI screenshot, receipt. Students must choose caption / VQA / OCR and justify. Compare BLEU-ish n-gram overlap vs human “is it true?”

Bridge: Preview video understanding as captioning + STT + temporal sampling.

Recap: Image captioning is vision→language generation. Continue with Video understanding.