GPT-1 proved generative pretraining; GPT-2 (Radford et al., 2019) scaled data and model size and popularized zero-shot task performance via natural language prompts—without task-specific fine-tuning heads for many demos.
It sits between GPT-1’s fine-tune paradigm and GPT-3’s few-shot in-context learning, and remains a practical open checkpoint for learning prompts and completions.
Learning Objectives
By the end of this lesson, students should be able to:
- Locate GPT-2 on the family timeline and list size variants (117M–1.5B).
- Explain zero-shot prompting as framing tasks as text completion.
- Describe byte-level BPE and long-context generation behavior.
- Generate text with Hugging Face
generate. - Discuss staged release / misuse concerns historically around GPT-2.
- Contrast zero-shot (GPT-2 story) with few-shot (GPT-3 story).
GPT-2 is a larger decoder-only LM (up to 1.5B parameters) trained on WebText-scale data, demonstrating that scaling autoregressive LMs improves zero-shot task transfer when tasks are phrased as language modeling prompts.
Timeline Checkpoint
Pretrain + fine-tune.
Scale; zero-shot prompts.
Few-shot ICL.
Chat & multimodal.
| Variant | Layers | d_model | ~Params |
|---|---|---|---|
| GPT-2 Small | 12 | 768 | 117M |
| GPT-2 Medium | 24 | 1024 | 345M |
| GPT-2 Large | 36 | 1280 | 762M |
| GPT-2 XL | 48 | 1600 | 1.5B |
Zero-Shot Framing
Instead of a classification head, write a prompt the LM can continue: “Translate to French: sea otter =>” and sample the completion. Quality varies, but the interface shift—tasks as text—is the conceptual leap.
Strengths and Tradeoffs
Strengths
- Open, well-documented teaching models.
- Clear scaling narrative across sizes.
- Prompt-as-program intuition.
Tradeoffs
- Zero-shot still unreliable vs. fine-tunes.
- Hallucinations and bias amplify with fluency.
- 1.5B is tiny vs. modern APIs.
“GPT-2 invented prompting.” Humans always conditioned LMs; GPT-2 made zero-shot task transfer via prompts a headline result at larger scale.
Knowledge Check
- Short Answer: What is GPT-2 XL’s approximate size? Answer: About 1.5 billion parameters.
- True/False: GPT-2 emphasized zero-shot task demos via prompts. Answer: True.
- Multiple Choice: GPT-2 tokenizer style: (a) byte-level BPE, (b) only characters, (c) pixels. Answer: (a).
- Short Answer: How does zero-shot casting work? Answer: Phrase the task so the answer is a natural continuation.
- True/False: GPT-2 is encoder-only like BERT. Answer: False.
- Multiple Choice: Relative to GPT-3, GPT-2 few-shot ICL is: (a) the main GPT-2 headline, (b) less central than zero-shot scaling story, (c) about CNNs. Answer: (b).
- Short Answer: Name one GPT-2 size between Small and XL. Answer: Medium or Large.
- Short Answer: Which HF method continues a prompt? Answer: model.generate (...).
- Multiple Choice: Staged release debates around GPT-2 concerned: (a) misuse risk vs. openness, (b) only GPU brands, (c) CSS layouts. Answer: (a).
- True/False: Larger GPT-2 variants generally produce more coherent long text than smaller ones. Answer: True (typically).
Key Takeaways
- GPT-2 scales decoder-only LMs and showcases zero-shot prompts.
- Size ladder from 117M to 1.5B teaches scaling intuition.
- Tasks become text completions.
- Still open and ideal for local experiments.
- Next: GPT-3.
Hands-on idea: Compare greedy vs. top-p samples from the same GPT-2 prompt.
Discussion prompt: When is zero-shot prompting enough vs. collecting labels to fine-tune?
Recap: GPT-2 made scale and zero-shot prompting famous. Continue with GPT-3.