CLIP — Module 7.3 | Codextroop

How This Lesson Fits the Module

The Vision Transformer gave us a powerful image encoder, but it still classified into a fixed list of labels. CLIP (Contrastive Language–Image Pre-training, OpenAI, 2021) removes that limit. By training an image encoder and a text encoder together on 400M image–text pairs, CLIP learns a shared embedding space—enabling zero-shot classification with arbitrary text labels, no fine-tuning required.

ViT / CNNs — classify into a fixed label set ↓ CLIP (2021) — align images and text; recognize any text label zero-shot ↓ Enables: open-vocabulary detection, retrieval, and prompt-driven vision like SAM

Learning Objectives

By the end of this lesson, students should be able to:

Explain CLIP’s dual-encoder (image + text) contrastive design.
Describe the shared embedding space and cosine-similarity matching.
Perform zero-shot classification with text prompts.
Explain why CLIP generalizes to unseen categories.
Run CLIP inference via transformers or open_clip.
Recognize CLIP’s limitations and biases.

Definition — CLIP

CLIP jointly trains an image encoder (a ViT or ResNet) and a text encoder (a transformer) so that matching image–text pairs land close together in a shared vector space and mismatched pairs are pushed apart. This contrastive objective lets you classify an image by comparing it to the text embeddings of candidate labels.

Contrastive Training

Given a batch of N image–text pairs, CLIP computes an N×N similarity matrix between all image and text embeddings. It maximizes similarity on the N correct (diagonal) pairs and minimizes it on the N²−N incorrect pairs. After training, an image and its true caption embed nearby—so any concept expressible in words becomes a possible class.

Zero-Shot Classification

To classify without training: embed the image once, embed each candidate label as a prompt (“a photo of a {label}”), and pick the label with highest cosine similarity. Change the label set at inference time—no retraining.

# pip install transformers
import torch
from transformers import CLIPProcessor, CLIPModel

model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

labels = ["a photo of a cat", "a photo of a dog", "a photo of a car"]
inputs = processor(text=labels, images=pil_image,
                   return_tensors="pt", padding=True)

with torch.no_grad():
    outputs = model(**inputs)
    logits_per_image = outputs.logits_per_image     # image-vs-text scores
    probs = logits_per_image.softmax(dim=1)         # zero-shot probabilities

for label, p in zip(labels, probs[0].tolist()):
    print(f"{label}: {p:.3f}")

ViT Tie-In The default CLIP image encoder is a Vision Transformer—patches, self-attention, [CLS] token. CLIP simply projects its output into a space shared with language.

What CLIP Unlocks

Zero-shot classification
Image–text retrieval / search
Open-vocabulary labels
Strong transfer features

What It Powers

Text-to-image models (guidance)
Open-vocabulary detection
Content moderation / tagging
Multimodal foundation models

Common Misconception: “CLIP was trained on labeled classification datasets.”

Reality: CLIP learned from ~400M noisy image–caption pairs scraped from the web—natural language supervision, not curated class labels. That is precisely why it generalizes to open-ended text categories.

Critical Mistake — Ignoring Prompt Engineering

CLIP’s zero-shot accuracy is sensitive to how labels are phrased. Raw class names (“cat”) underperform prompt templates (“a photo of a cat”), and ensembling several templates helps further. Also remember CLIP inherits web-scale biases—audit before high-stakes use.

Knowledge Check

Short Answer: What do the two CLIP encoders process? Answer: Images and text.
True/False: CLIP enables zero-shot classification. Answer: True.
Multiple Choice: CLIP’s training objective is: (a) cross-entropy on 1000 classes, (b) contrastive image–text matching, (c) pixel reconstruction, (d) next-token prediction. Answer: (b).
Short Answer: How do you classify an image with CLIP at inference? Answer: Compare its embedding to text-prompt embeddings via cosine similarity.
True/False: CLIP was trained on curated labeled datasets. Answer: False—web image–text pairs.
Multiple Choice: A common CLIP image encoder is a: (a) LSTM, (b) ViT, (c) decision tree, (d) U-Net. Answer: (b).
Short Answer: Why does prompt phrasing matter? Answer: Zero-shot accuracy depends on how labels are worded (prompt templates help).
True/False: You must fine-tune CLIP to change its label set. Answer: False—just change the text prompts.
Multiple Choice: The shared space lets CLIP measure image–text match via: (a) cosine similarity, (b) IoU, (c) BLEU, (d) FID. Answer: (a).
Short Answer: Name one downstream capability CLIP powers. Answer: Image retrieval, open-vocabulary detection, text-to-image guidance (any).

Key Takeaways

CLIP aligns image and text encoders in one shared embedding space.
Contrastive training on web image–text pairs teaches broad concepts.
Zero-shot classification works by comparing images to text-prompt embeddings.
It unlocks open-vocabulary vision, retrieval, and multimodal systems.
Next: the Vol 07 capstone—promptable segmentation with SAM.

Trainer’s Guide

Demo: Classify the same image with different label sets on the fly—students witness zero-shot flexibility no CNN can match.

Exercise: Have students compare raw class names vs prompt templates vs template ensembles and measure the accuracy gains from prompt engineering.

Almost There We can now recognize open-ended concepts from language. The final model of Vol 07 makes segmentation promptable and universal. Continue to the capstone: SAM (Segment Anything).