Module 11.2 covered encoder-only NLU (BERT family). Module 11.3 opens the generative line: GPT-1 (Radford et al., 2018) showed that a decoder-only Transformer pretrained with autoregressive language modeling transfers to many NLP tasks via fine-tuning—the seed of today’s LLMs.
The lineage continues through GPT-2, GPT-3, GPT-3.5, GPT-4, and forward-looking themes in GPT-5.
Learning Objectives
By the end of this lesson, students should be able to:
- Define GPT-1 as unsupervised pretrain + supervised fine-tune on a decoder Transformer.
- Place GPT-1 on the GPT family timeline.
- Explain causal self-attention vs. BERT bidirectionality.
- Describe task formatting via start/delimiter tokens for fine-tuning.
- Sketch a causal LM forward pass in PyTorch / HF.
- State GPT-1’s approximate scale (~117M) relative to later GPT models.
GPT-1 (Generative Pre-trained Transformer) is OpenAI’s 2018 decoder-only Transformer trained with left-to-right language modeling on BooksCorpus, then fine-tuned on labeled downstream tasks with minimal architecture changes.
GPT Family Timeline (Start)
~117M; pretrain + fine-tune.
Scale + zero-shot demos.
Few-shot in-context learning.
Alignment, multimodality.
Architecture Ideas
Decoder-only stack
- Transformer decoder blocks
- Causal attention
- Tied LM head over vocab
Pretraining
- Maximize P(token | past)
- Large unlabeled text
- No MLM masks
Fine-tuning
- Task-specific linear head
- Formatted examples
- Still LM-friendly inputs
| Property | GPT-1 (approx.) | BERT-Base |
|---|---|---|
| Attention | Causal | Bidirectional |
| Objective | Next-token LM | MLM (+ NSP) |
| Params | ~117M | ~110M |
| Sweet spot | Generation + transfer | NLU encoders |
Minimal Causal LM Sketch
Note: Original GPT-1 weights are less commonly used today; GPT-2 checkpoints illustrate the same decoder-only API students will use throughout this module.
“GPT-1 already did ChatGPT-style instruction following.” GPT-1 emphasized unsupervised pretraining plus supervised fine-tuning on NLP benchmarks—not chat RLHF. Instruction-tuned assistants arrive much later (GPT-3.5 era).
Strengths and Tradeoffs
Strengths
- Unified generative pretrain recipe.
- Natural text generation.
- Transfer with light fine-tuning.
Tradeoffs
- No bidirectional context for NLU spans.
- Small by modern standards.
- Still needs labeled fine-tunes for many tasks.
Knowledge Check
- Short Answer: What does GPT stand for in this lineage? Answer: Generative Pre-trained Transformer.
- True/False: GPT-1 uses bidirectional MLM like BERT. Answer: False—causal LM.
- Multiple Choice: GPT-1’s stage recipe is: (a) only RLHF, (b) unsupervised LM pretrain then supervised fine-tune, (c) only k-means. Answer: (b).
- Short Answer: Rough GPT-1 parameter count? Answer: About 117 million.
- True/False: Causal attention lets tokens see future positions. Answer: False.
- Multiple Choice: Compared with BERT, GPT-1 is stronger at: (a) open-ended generation, (b) only image nets, (c) sorting algorithms only. Answer: (a).
- Short Answer: Which module lecture details decoder-only stacks? Answer: Decoder Only.
- Short Answer: Name the year GPT-1 appeared. Answer: 2018.
- Multiple Choice: Next in the timeline after GPT-1: (a) GPT-2, (b) AlexNet, (c) Word2Vec only. Answer: (a).
- True/False: GPT-1 already relied primarily on few-shot prompting instead of fine-tuning. Answer: False—that theme grows with GPT-3.
Key Takeaways
- GPT-1 established decoder-only generative pretraining for NLP transfer.
- Causal LM + fine-tune, contrasting BERT’s MLM encoders.
- ~117M parameters; BooksCorpus-scale pretraining.
- Starts the GPT timeline continued in later lectures.
- Next: GPT-2.
Hands-on idea: Draw BERT vs. GPT-1 attention masks side by side for a 5-token sentence.
Discussion prompt: Why might generative pretraining help even classification tasks after fine-tuning?
Recap: GPT-1 planted the generative Transformer recipe. Continue with GPT-2.