CLIP — Module 16.1

How This Lesson Fits the Module & Volume

ViT encodes images. CLIP aligns those visual embeddings with text embeddings so one cosine space supports zero-shot labels, retrieval, and generator conditioning. You met CLIP as a vision model in Vol. 07 CLIP; the text geometry sits on Vol. 09 sentence embeddings and cosine search (Vol. 14). Here CLIP is a multimodal capability: image–text lock, not a new contrastive-math course.

Captioning (next-but-one) generates sentences; CLIP scores or retrieves them. 16.3 generators often condition on CLIP/T5 text—do not confuse conditioning with the 16.3 product list.

Learning Objectives

By the end of this lesson, students should be able to:

Explain CLIP’s dual encoder and contrastive image–text space.
Perform zero-shot classification and text–image retrieval.
Contrast CLIP scoring with image captioning (generation).
Run CLIP inference in Python and interpret cosine scores.
List limitations: compositionality, fine text, OCR-ish reading, bias.
Connect CLIP to Vol. 07/09/14 and to 16.3 prompt conditioning.

Definition

CLIP (Contrastive Language–Image Pre-training) jointly trains an image encoder (often a ViT) and a text encoder so matching pairs are close in a shared vector space and mismatches are far. At inference you embed images and free-form text once, then compare with cosine similarity—no task-specific classifier head required.

Dual Encoder Recap

Tower	Input	Heritage
Image encoder	Pixels → vector	ViT / CNN (Vol. 07, Vol. 10)
Text encoder	Tokens → vector	Transformer + Vol. 09 embeddings
Loss	Batch contrastive (InfoNCE-style)	Vol. 07 CLIP lecture
Match	Cosine similarity	Vol. 14 cosine

What CLIP Is For (and Not)

CLIP does well

Zero-shot labels via prompts
Image–text retrieval / search
Open-vocab filters, dataset hygiene

Use another capability

Exact text in pixels → OCR
A full sentence description → captioning
New pixels → image gen / 16.3

Agent pattern

Embed once, cache vectors
Retrieve then maybe caption
Do not dump CLIP scores as “the answer”

Practical Zero-Shot + Retrieval

import torch
import torch.nn.functional as F
from transformers import CLIPProcessor, CLIPModel

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
proc = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
model.eval()

def zero_shot(image, labels: list[str]) -> str:
    prompts = [f"a photo of {x}" for x in labels]
    inputs = proc(text=prompts, images=image, return_tensors="pt", padding=True)
    with torch.no_grad():
        out = model(**inputs)
        probs = out.logits_per_image.softmax(dim=-1)[0]
    return labels[int(probs.argmax())]

def text_to_image_scores(query: str, images: list) -> torch.Tensor:
    inputs = proc(text=[query], images=images, return_tensors="pt", padding=True)
    with torch.no_grad():
        img_f = model.get_image_features(pixel_values=inputs["pixel_values"])
        txt_f = model.get_text_features(input_ids=inputs["input_ids"],
                                        attention_mask=inputs["attention_mask"])
    img_f = F.normalize(img_f, dim=-1)
    txt_f = F.normalize(txt_f, dim=-1)
    return (img_f @ txt_f.T).squeeze(-1)  # cosine scores
# Cache image_f in a vector DB (Vol. 14) for production search.

Limits You Must Teach

Strengths

Open vocabulary without retraining
Natural language queries
Strong coarse semantics

Weak spots

Counting, negation, binding (“red cube on blue”)
Tiny text / exact OCR strings
Dataset bias & spurious correlations

Common Misconception

“CLIP captions images.” CLIP can rank candidate captions or labels; it does not decode a sentence unless you add a generator (captioning lecture) or an LLM. Also: Vol. 09 embeddings alone are text-only—CLIP is the lock that puts images into that kind of space. Vol. 07 taught the model; 16.1 teaches the multimodal capability you will call from agents.

Knowledge Check

Short Answer: What two encoders does CLIP train? Answer: An image encoder and a text encoder (shared embedding space).
True/False: CLIP zero-shot needs a new softmax head per dataset. Answer: False—you embed label prompts and compare cosines.
Multiple Choice: Exact receipt totals should use: (a) OCR, (b) CLIP cosine only, (c) TTS. Answer: (a).
Short Answer: Which Vol. 09 idea is the text-side intuition for CLIP? Answer: Sentence / text embeddings (cosine geometry).
True/False: Image generation conditioning on CLIP text is the same as Module 16.3’s vendor catalog. Answer: False—conditioning is a capability; 16.3 lists products.
Multiple Choice: Retrieval with CLIP is closest to: (a) Vol. 14 vector search, (b) lip sync, (c) diarization. Answer: (a).
Short Answer: Name one CLIP failure mode. Answer: Compositionality, counting, negation, tiny text, bias (any one).
True/False: ViT is sufficient to do CLIP without a text tower. Answer: False.
Multiple Choice: Vol. 07 CLIP vs 16.1 CLIP: (a) vision-model lecture vs multimodal capability, (b) TTS vs STT, (c) k-means vs PCA. Answer: (a).
Short Answer: Next lecture? Answer: Speech-to-text.

Key Takeaways

CLIP = contrastive image–text space (ViT + text encoder).
Use it to score, retrieve, and zero-shot label—not to OCR or caption by itself.
Vol. 07 / Vol. 09 / Vol. 14 supply the math; 16.1 supplies the agent capability.
16.3 models may use CLIP-like conditioning; that is not the catalog.
Next: Speech-to-text.

Trainer’s Guide

Lab: Same 12 images: (1) ViT k-NN, (2) CLIP text query, (3) OCR on any with writing. Students write which capability won.

Prompt craft: Show “a photo of X” vs bare labels; connect to how 16.3 prompts are just text embeddings at scale.

Recap: CLIP locks vision to language for retrieval and zero-shot. Continue with Speech-to-text.