History of Transformers — Module 10.2

How This Lesson Fits the Module & Volume

Module 10.1 built the primitives—attention, Query/Key/Value, scaled dot-product attention, multi-head attention, positional encoding, LayerNorm, FFN, and residuals.

Module 10.2 assembles those parts into the full Transformer. This opening lecture is the story: how seq2seq RNNs, Bahdanau attention, and the 2017 paper “Attention Is All You Need” led to BERT, GPT, and Vision Transformers.

Learning Objectives

By the end of this lesson, students should be able to:

Trace the path from RNN seq2seq to attention-augmented models to the Transformer.
State what problem “Attention Is All You Need” solved relative to recurrence.
Distinguish encoder-only (BERT), decoder-only (GPT), and encoder–decoder lineages.
Place Vision Transformers (ViT) in the same architectural family as NLP Transformers.
Map Module 10.1 building blocks to the historical timeline.
Preview the Module 10.2 lecture sequence from history through ViT.

Definition

The Transformer is a neural sequence architecture introduced by Vaswani et al. (2017) that replaces recurrence and convolution with stacked self-attention and position-wise feed-forward networks, enabling full parallelization over sequence length during training.

Timeline: From Seq2Seq to Today

RNN Seq2Seq

Encode then decode with hidden states (2014).

Attention

Bahdanau/Luong soft alignment over encoder states.

Transformer

Attention Is All You Need (2017).

BERT / GPT / ViT

Encoder, decoder, and vision lineages.

Why RNNs Hit a Wall

Early neural machine translation used an encoder RNN to compress a source sentence into a fixed vector, then a decoder RNN to generate the target. Long sentences lost information in that bottleneck; training was sequential (timestep t depends on t−1), so GPUs underutilized sequence length.

Attention (Bahdanau 2015) let the decoder look at all encoder states via soft weights—exactly the Q/K/V idea from Module 10.1. The Transformer’s leap: drop the RNN entirely and use attention for both mixing tokens and conditioning the decoder.

Era	Core idea	Limitation
RNN seq2seq	Recurrent hidden state over time	Sequential training; long-range fade
Attn + RNN	Soft alignment over encoder states	Still recurrent backbone
Transformer	Self-attention + FFN stacks	Quadratic attention cost in length
BERT / GPT / ViT	Specialize encoder / decoder / patches	Data and compute hungry

Three Lineages After 2017

Encoder-only (BERT)

Bidirectional self-attention.
Masked language modeling pretraining.
Strong for understanding / classification.

Decoder-only (GPT)

Causal (masked) self-attention.
Next-token prediction at scale.
Foundation of modern LLMs (Vol. 11).

Encoder–Decoder / ViT

Original NMT stack; T5, BART.
ViT: image patches as tokens.
Same block recipe, new inputs.

Code: Historical Sketch in Shapes

This snippet shows how Module 10.1 pieces already imply the modern stack in PyTorch shapes.

import torch
from torch import nn

# Toy "attention era" shapes: batch=2, src_len=5, tgt_len=4, d_model=16
B, S, T, D = 2, 5, 4, 16
enc_states = torch.randn(B, S, D)   # what an RNN encoder would emit
dec_query  = torch.randn(B, T, D)   # decoder query at each target step

# Bahdanau-style scores: each target position attends over all source positions
scores = torch.bmm(dec_query, enc_states.transpose(1, 2)) / (D ** 0.5)  # (B, T, S)
weights = scores.softmax(dim=-1)
context = torch.bmm(weights, enc_states)  # (B, T, D)

# Transformer insight: same math, but Q/K/V from the *same* sequence (self-attn)
x = enc_states
q = k = v = x
self_scores = torch.bmm(q, k.transpose(1, 2)) / (D ** 0.5)
self_out = torch.bmm(self_scores.softmax(-1), v)
print(context.shape, self_out.shape)  # torch.Size([2, 4, 16]) torch.Size([2, 5, 16])

Strengths of the Historical Shift

What Transformers Unlocked

Parallel training over the full sequence length.
Direct long-range dependencies via attention.
One family for MT, LM, vision, and multimodal.

Ongoing Costs

O(n²) attention memory/compute in sequence length.
Needs positional signals (encoding or embedding).
Large models demand careful optimization and data.

Common Misconception

“Transformers replaced attention.” The opposite: Transformers are attention-centric. They replaced recurrence as the primary sequence mixer. Module 10.1’s attention math is the heart of Module 10.2’s architecture.

Module 10.2 Roadmap

Next you will define the Transformer, walk the full stack, open encoder and decoder blocks, deepen MHSA and masked attention, contrast positional embeddings with 10.1 encodings, revisit skip connections, and cap with the Vision Transformer toward Vol. 11.

Knowledge Check

Short Answer: What bottleneck did classic RNN seq2seq suffer from? Answer: Compressing the whole source into one fixed vector (information bottleneck).
True/False: Bahdanau attention removed the need for any encoder states beyond a single vector. Answer: False—it uses soft weights over all encoder states.
Multiple Choice: “Attention Is All You Need” primarily replaced: (a) softmax, (b) recurrence/convolution as the main mixer, (c) embeddings. Answer: (b).
Short Answer: Name one encoder-only and one decoder-only lineage. Answer: BERT (encoder-only); GPT (decoder-only).
True/False: Vision Transformers use a completely different attention formula than NLP Transformers. Answer: False—same self-attention idea on patch tokens.
Multiple Choice: A key training advantage of Transformers over RNNs is: (a) smaller vocab, (b) parallelization over sequence length, (c) no need for GPUs. Answer: (b).
Short Answer: Which Module 10.1 topic supplies Q, K, and V? Answer: Attention / Query, Key, Value lectures.
Short Answer: Why do Transformers still need positional information? Answer: Self-attention is permutation-equivariant without position signals.
Multiple Choice: GPT-style models rely on: (a) bidirectional MLM only, (b) causal masked self-attention, (c) CNN backbones. Answer: (b).
True/False: Module 10.2 builds the full architecture from Module 10.1 primitives. Answer: True.

Key Takeaways

History: RNN seq2seq → attention → Transformer → BERT/GPT/ViT.
The 2017 paper kept attention and dropped recurrence as the main sequence engine.
Lineages specialize the same block: encoder-only, decoder-only, or both—plus vision patches.
Module 10.1 math is the toolkit; Module 10.2 is the assembled machine.
Next: Transformer definition and high-level structure.

Trainer’s Guide

Hands-on idea: Sketch a 2014 seq2seq diagram vs. a 2017 Transformer on the board; have students circle where recurrence disappeared and where self-attention appeared.

Discussion prompt: If attention already existed in 2015, what made “Attention Is All You Need” a paradigm shift rather than an incremental tweak?

Recap: Transformers grew from seq2seq pain points and attention; Module 10.2 now builds the architecture. Continue with Transformer.