Vision Transformer (ViT) — Module 16.1

How This Lesson Fits the Module & Volume

Module 16.1 is about modalities, but several capabilities share one encoder: the Vision Transformer. You studied ViT as architecture in Vol. 07 and as the Vol. 10 capstone in Vol. 10 ViT. Here we treat ViT as the workhorse visual backbone behind CLIP, captioning, many VLMs, and even spectrogram audio—not a second full transformer course.

Next lecture (CLIP) adds the text tower. Image generators in 16.3 and video models in 16.4 often use ViT/DiT variants; 16.1 just names why.

Learning Objectives

By the end of this lesson, students should be able to:

Recap ViT: patches, CLS, positional embeddings, encoder stack.
Contrast CNN inductive bias (Vol. 07) with ViT global attention (Vol. 10).
Explain where ViT sits in CLIP, captioning, OCR, and VLMs.
Run a ViT classifier forward pass in PyTorch/transformers.
Discuss compute: resolution, patch size, and video token explosion.
Link ViT skills to multimodal agents without retaking Volume 10.

Definition

A Vision Transformer (ViT) splits an image into fixed patches, linearly embeds each patch, adds positional embeddings (and usually a [CLS] token), and runs a Transformer encoder. The resulting tokens are visual features for classification, retrieval, captioning, or as a frozen tower in CLIP.

Recap: Pixels to Tokens

Step	Role	Volume link
Patchify	P×P cells → sequence	Vol. 10 ViT
Linear embed + PE	d_model vectors + order	Positional embedding
MHSA encoder	Global mix of patches	MHSA
Head	Class / features / decode	Task-specific (16.1 capabilities)

CNN vs ViT in Multimodal Products

CNN (Vol. 07)

Local filters, translation bias
Strong on small data / mobile
Still common in OCR detectors

ViT (Vol. 10)

Global attention from layer 1
Scales with data/compute
Default CLIP / VLM encoder

Hybrids

CNN stem + ViT body
Windowed / Swin attention
Video: tubelets / time tokens

Where ViT Shows Up in 16.1

Capability	ViT’s job
CLIP	Image tower → embedding
Captioning	Visual tokens for a text decoder
OCR	Recognize crops / full-page encoders
Video understanding	Per-frame or tubelet encoder
Image / video gen	DiT = transformer on latent patches

Practical Inference

# ViT as a vision backbone (features, not just ImageNet labels)
import torch
from PIL import Image
from transformers import ViTImageProcessor, ViTModel

processor = ViTImageProcessor.from_pretrained("google/vit-base-patch16-224")
model = ViTModel.from_pretrained("google/vit-base-patch16-224")
model.eval()

def vit_cls(path: str) -> torch.Tensor:
    img = Image.open(path).convert("RGB")
    inputs = processor(images=img, return_tensors="pt")
    with torch.no_grad():
        out = model(**inputs)
    # CLS token: drop-in visual embedding for retrieval or a linear head
    return out.last_hidden_state[:, 0, :]  # [1, hidden]

# For video: sample keyframes, encode each, then temporal pool / attend.
# Token count ≈ (H/P)*(W/P); doubling resolution ~4x tokens — plan 16.1 video budgets.

When Not to Reach for ViT

ViT shines

Large pretrain, flexible resolution
Multimodal towers (CLIP, VLMs)
Need global context (charts, UI)

CNN still wins

Tiny models on-device
Dense prediction with little data
Classic detectors in OCR pipelines

Common Misconception

“ViT replaced CNNs, so Volume 07 is obsolete.” Production vision still mixes both. OCR detectors, MobileNet-class classifiers, and many video backbones remain convolutional. ViT is the dominant multimodal encoder, not a ban on convolution. Also: ViT is not CLIP by itself—CLIP adds a text encoder and contrastive loss (next lecture; Vol. 07 CLIP + Vol. 09 embeddings).

Knowledge Check

Short Answer: How does ViT turn an image into a transformer input? Answer: Split into patches, embed, add positions (± CLS), encode.
True/False: This 16.1 lecture replaces Volume 10’s full ViT derivation. Answer: False—it reapplies ViT as a multimodal backbone.
Multiple Choice: CLIP’s image tower is commonly a: (a) ViT, (b) TTS vocoder, (c) k-means. Answer: (a).
Short Answer: Which volume introduced CNN locality bias? Answer: Volume 07.
True/False: Doubling image side length roughly quadruples ViT patch tokens. Answer: True (for fixed patch size).
Multiple Choice: DiT-style generators relate to ViT because they: (a) attend over latent patches, (b) do STT, (c) parse PDFs. Answer: (a).
Short Answer: Why do video pipelines sample frames before ViT? Answer: Token/compute explosion across time.
True/False: CNNs are unused in modern OCR. Answer: False—detectors often remain CNN-based.
Multiple Choice: Vol. 10 positional embeddings in ViT encode: (a) patch order/layout, (b) speaker identity, (c) WER. Answer: (a).
Short Answer: Next lecture? Answer: CLIP.

Key Takeaways

ViT is the default visual encoder for multimodal 16.1 capabilities.
Architecture recap: patches + PE + encoder (Vol. 07 / Vol. 10).
CNNs remain useful; ViT did not delete Volume 07.
Watch token counts for high-res and video.
Next: CLIP adds aligned text.

Trainer’s Guide

Mini-lab: Extract ViT CLS vectors for 10 images; nearest-neighbor retrieve. Then compare to CLIP (next class) on the same set.

Review: 5-minute Vol. 10 encoder-block sketch so students remember MHSA vs CNN receptive fields.

Recap: ViT is 16.1’s visual backbone, not a new transformer textbook. Continue with CLIP.