MLM only supervises a subset of tokens (~15%), so compute per token of learning signal is low. ELECTRA (Clark et al., 2020) replaces MLM with a discriminative task: detect which tokens a small generator replaced. The discriminator sees every position—sample-efficient pretraining that often matches larger MLM models at lower cost.
It closes the “better objectives / cheaper training” arc before Sentence-BERT specializes embeddings for similarity.
Learning Objectives
By the end of this lesson, students should be able to:
- Contrast generative MLM with ELECTRA’s replaced-token detection (RTD).
- Describe the generator–discriminator setup.
- Explain why RTD yields denser learning signals.
- Load an ELECTRA model in Hugging Face for classification.
- Relate ELECTRA’s discriminator to downstream fine-tuning (generator discarded).
- Compare compute efficiency vs. BERT/RoBERTa pretraining.
ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately) trains a small MLM generator to propose token replacements and a discriminator to predict, for every position, whether the token is original or replaced. After pretraining, typically only the discriminator is fine-tuned.
Replaced Token Detection
Mask some tokens like MLM.
Small MLM fills proposals.
Swap masks with sampled tokens.
Discriminator labels real vs. fake per position.
| Aspect | BERT MLM | ELECTRA RTD |
|---|---|---|
| Task type | Generative (vocab softmax) | Binary per position |
| Supervised positions | ~15% masked | All tokens |
| Auxiliary model | None | Small generator |
| Deployed weights | Full encoder | Discriminator encoder |
Why It Works
Binary detection is cheaper than full-vocab softmax at every masked spot, and the learning signal covers the entire sequence. Plausible generator replacements make the discriminator solve a hard, language-sensitive problem—not a trivial “spot [MASK]” cue.
Strengths and Tradeoffs
Strengths
- Strong results at smaller compute budgets.
- Dense supervision on every token.
- Clear efficiency story for pretraining.
Tradeoffs
- Two networks during pretraining (complexity).
- Generator quality affects difficulty.
- Less “fill-in-the-blank” generative use than MLM.
“ELECTRA is a GAN.” The paper discusses adversarial ideas, but training is not a classic minimax GAN loop. Think: generator proposes corruptions; discriminator does efficient discriminative LM pretraining.
Knowledge Check
- Short Answer: What does RTD stand for? Answer: Replaced Token Detection.
- True/False: ELECTRA’s discriminator is supervised on every token position. Answer: True.
- Multiple Choice: After pretraining you typically fine-tune: (a) only the generator, (b) the discriminator, (c) a diffusion U-Net. Answer: (b).
- Short Answer: Why is MLM sample-inefficient? Answer: Loss applies mainly to the masked minority of tokens.
- True/False: ELECTRA’s generator is usually smaller than the discriminator. Answer: True (common setup).
- Multiple Choice: Discriminator output per token is essentially: (a) binary real/replaced, (b) full vocab softmax always, (c) a pixel. Answer: (a).
- Short Answer: Name one efficiency benefit vs. BERT. Answer: Better accuracy per pretraining FLOP / denser loss.
- Short Answer: Is ELECTRA typically encoder-only? Answer: Yes.
- Multiple Choice: ELECTRA is closest in goal to: (a) chat RLHF, (b) efficient NLU pretraining, (c) speech codecs. Answer: (b).
- True/False: ELECTRA training is identical to training StyleGAN. Answer: False.
Key Takeaways
- ELECTRA learns by detecting generator replacements on all positions.
- Denser loss than MLM improves compute efficiency.
- Fine-tune the discriminator encoder for downstream tasks.
- Not a classic GAN despite generator/discriminator naming.
- Next: Sentence-BERT for sentence embeddings.
Hands-on idea: Sketch a 5-token toy sequence; mark which positions get RTD labels vs. MLM labels.
Discussion prompt: When is a discriminative pretraining objective preferable to generative MLM?
Recap: ELECTRA makes pretraining denser and cheaper. Continue with Sentence-BERT.