GPT-2 scaled to 1.5B; GPT-3 (Brown et al., 2020) jumped to 175B and centered few-shot in-context learning: put examples in the prompt, and the model adapts without weight updates. That idea reshapes how engineers use LMs and leads into chat-aligned GPT-3.5.
Learning Objectives
By the end of this lesson, students should be able to:
- Place GPT-3 on the timeline and cite ~175B as the flagship size.
- Define zero-, one-, and few-shot prompting.
- Explain in-context learning vs. fine-tuning.
- Discuss context window limits as a practical constraint.
- Relate API completions to Module 11.1 sampling concepts.
- Note compute/data scale as the main “architecture” story (still decoder-only).
GPT-3 is a family of autoregressive decoder-only Transformers (flagship 175B) trained at unprecedented scale, notable for strong few-shot performance where task demonstrations are provided only in the prompt context.
Timeline
Zero-shot emphasis.
Few-shot ICL at 175B.
Instruction / chat tuning.
Multimodal capability jump.
Zero-shot
- Instruction only
- No examples
- Hardest setting
One-shot
- Single demonstration
- Shows format
- Often big gains
Few-shot
- K examples in context
- No gradient steps
- GPT-3 headline
| Adaptation | Updates weights? | Where task lives |
|---|---|---|
| Fine-tuning | Yes | Parameter space |
| In-context learning | No | Prompt tokens |
| Retrieval + prompt | No (usually) | Fetched text + prompt |
Engineering Notes
Architecturally GPT-3 is still next-token prediction with causal attention—the leap is scale and the empirical discovery that large models meta-learn tasks from prompt patterns. Closed API access defined the product era; open reproductions and smaller public LMs teach the same ideas locally.
“Few-shot prompting fine-tunes the model.” Unless you run a training job, weights stay fixed; only activations over the context change. That is adaptation in context, not SGD fine-tuning.
Strengths and Tradeoffs
Strengths
- Flexible task switching via prompts.
- Strong few-shot results across NLP suites.
- Catalyzed the LLM product wave.
Tradeoffs
- Huge compute; limited open weights historically.
- Context length caps example count.
- Unaligned base models can be unsafe / unhelpful.
Knowledge Check
- Short Answer: Approximate GPT-3 flagship size? Answer: 175 billion parameters.
- True/False: Few-shot ICL updates gradients each example. Answer: False.
- Multiple Choice: GPT-3’s famous adaptation mode: (a) few-shot prompting, (b) only SVM kernels, (c) only batch norm. Answer: (a).
- Short Answer: Difference between zero-shot and few-shot? Answer: Few-shot includes demonstrations in the prompt.
- True/False: GPT-3 remains decoder-only / autoregressive. Answer: True.
- Multiple Choice: Context window limits: (a) how many examples you can fit, (b) CPU brand only, (c) CSS grid. Answer: (a).
- Short Answer: Name the 2020 GPT-3 paper’s core theme in three words. Answer: e.g., Language models few-shot / in-context learning.
- Short Answer: Does GPT-3 replace BERT for every classification deploy? Answer: No—encoders can still be cheaper/better for narrow tasks.
- Multiple Choice: Next curriculum step after GPT-3: (a) GPT-3.5, (b) only Vol. 02 vectors, (c) Dijkstra only. Answer: (a).
- True/False: In-context learning stores new tasks permanently in weights. Answer: False—it lives in the prompt.
Key Takeaways
- GPT-3 = scale + few-shot in-context learning.
- ~175B flagship; still causal decoder-only LM.
- Prompts carry task specifications and examples.
- Not the same as fine-tuning weights.
- Next: GPT-3.5.
Hands-on idea: Write one zero-shot and one 3-shot sentiment prompt; compare completions on a small local model.
Discussion prompt: What fails when few-shot examples are noisy or inconsistently formatted?
Recap: GPT-3 made in-context learning central. Continue with GPT-3.5.