Every model so far—from LeNet to Mask R-CNN—is built on convolution. The Vision Transformer (ViT) (Dosovitskiy et al., Google, 2020) broke that assumption: it splits an image into patches and feeds them to a pure transformer—the same self-attention architecture that revolutionized NLP. Given enough data, ViT matches or beats top CNNs, proving convolution is not the only path to strong vision.
Learning Objectives
By the end of this lesson, students should be able to:
- Explain how ViT converts an image into a sequence of patch embeddings.
- Describe self-attention and why it captures global context in one layer.
- Identify the roles of the [CLS] token, positional embeddings, and residual blocks.
- Contrast ViT’s inductive bias and data appetite with CNNs.
- Load a pretrained ViT via
timmortransformers. - Relate ViT’s residual structure back to ResNet.
A Vision Transformer divides an image into fixed-size patches (e.g., 16×16), linearly projects each patch into a token embedding, adds positional embeddings, prepends a learnable [CLS] token, and processes the sequence with standard transformer encoder blocks (multi-head self-attention + MLP, each wrapped in residual connections and layer norm). The [CLS] token’s final state is fed to a classifier head.
From Pixels to Tokens
A 224×224 image with 16×16 patches becomes a sequence of 196 tokens—analogous to 196 “words.” This is the crucial reframing: an image is a sentence of patches. Self-attention then lets every patch attend to every other patch, giving global context in a single layer—something a convolution only achieves after many stacked layers.
Self-attention computes, for each token, a weighted sum of all tokens’ values, where weights come from query–key similarity: Attention(Q, K, V) = softmax(QK⊤ / √d)V. It is the same mechanism used in language models; here the “tokens” are image patches instead of words. (You will study attention in depth when sequence models arrive in later volumes.)
| Aspect | CNN (e.g. ResNet) | Vision Transformer |
|---|---|---|
| Core operation | Convolution | Self-attention |
| Receptive field | Local, grows with depth | Global from layer 1 |
| Inductive bias | Strong (locality, translation) | Weak—learned from data |
| Data need | Works on modest data | Needs large data or pretraining |
| Year | 2015 | 2020 |
Loading a ViT in PyTorch
Reality: ViT only beats CNNs when trained on very large datasets (or pretrained then fine-tuned). On small datasets, CNNs’ built-in locality bias often wins. In practice, hybrid and CNN models remain competitive—architecture choice depends on data scale.
ViT lacks the locality prior that helps CNNs generalize from little data. Training it from scratch on a few thousand images usually underperforms a ResNet. Almost always start from pretrained weights and fine-tune—this is where ViT shines.
Knowledge Check
- Short Answer: How does ViT turn an image into transformer input? Answer: Split into patches, linearly embed each as a token.
- True/False: ViT uses convolution as its core operation. Answer: False—it uses self-attention.
- Multiple Choice: The [CLS] token is used to: (a) pad, (b) aggregate a representation for classification, (c) store pixels, (d) normalize. Answer: (b).
- Short Answer: Why are positional embeddings needed? Answer: Attention is permutation-invariant; they encode patch order/location.
- True/False: Self-attention gives global context in a single layer. Answer: True.
- Multiple Choice: Compared to CNNs, ViT has: (a) stronger locality bias, (b) weaker inductive bias, (c) no residuals, (d) fewer data needs. Answer: (b).
- Short Answer: What structural idea from ResNet appears inside every ViT block? Answer: Residual (skip) connections.
- True/False: ViT typically needs large-scale data or pretraining to beat CNNs. Answer: True.
- Multiple Choice: A 224×224 image with 16×16 patches yields how many patch tokens? (a) 49, (b) 196, (c) 256, (d) 1024. Answer: (b).
- Short Answer: Name one foundation model built on a ViT backbone. Answer: CLIP or SAM.
Key Takeaways
- ViT treats an image as a sequence of patch tokens processed by transformers.
- Self-attention captures global relationships from the first layer.
- It has weak inductive bias, so it needs large data or pretraining.
- Every block uses residual connections—the ResNet idea, generalized.
- Next: CLIP pairs a ViT with language for open-vocabulary vision.
Visualization: Plot ViT attention maps for the [CLS] token—students see the model attending to the actual object, not the background.
Experiment: Fine-tune ViT vs ResNet-50 on a 2,000-image dataset, then on a 200,000-image dataset; students observe how ViT’s advantage grows with data.