Masked Language Modeling — Module 11.2

How This Lesson Fits the Module & Volume

You have seen the BERT family variants; this lecture isolates the core pretraining objective that made bidirectional encoders possible: Masked Language Modeling. It links Module 11.1’s next-token prediction (causal LMs) to encoder pretraining, and prepares the contrast with NSP.

Learning Objectives

By the end of this lesson, students should be able to:

Define MLM and the classic 15% masking schedule (80/10/10).
Explain how MLM enables full bidirectional attention without leaking answers.
Contrast MLM with causal next-token prediction.
Implement a conceptual MLM loss in PyTorch (masked positions only).
Run a Hugging Face fill-mask pipeline.
Discuss MLM limitations that motivated ELECTRA / RoBERTa changes.

Definition

Masked Language Modeling randomly hides tokens in the input (often replacing them with a mask token) and trains the model to predict the original identities from bidirectional context. Loss is computed on the corrupted positions.

The Classic BERT Masking Recipe

Select ~15% of WordPiece tokens. Of those:

Fraction of selected	Replacement	Rationale
80%	`[MASK]`	Main cloze training signal
10%	Random vocabulary token	Reduce mask-token train/test mismatch
10%	Unchanged token	Bias model toward real inputs

Causal LM

Predict token t from < t
Natural for generation
No future context

MLM

Predict masked from both sides
Natural for NLU encoders
Sparse supervision

Dynamic MLM

Resample masks each epoch
RoBERTa-style
Harder memorization

Loss Sketch

import torch
import torch.nn.functional as F

# logits: (batch, seq, vocab), labels: (batch, seq) with -100 = ignore
def mlm_loss(logits, labels):
    return F.cross_entropy(
        logits.view(-1, logits.size(-1)),
        labels.view(-1),
        ignore_index=-100,
    )

logits = torch.randn(2, 8, 1000)
labels = torch.full((2, 8), -100)
labels[0, 3] = 42  # only position 3 supervised in example
print(mlm_loss(logits, labels))

Hugging Face Fill-Mask

from transformers import pipeline
unmasker = pipeline("fill-mask", model="bert-base-uncased")
print(unmasker("The capital of France is [MASK].")[:3])

Common Misconception

“MLM trains on every token equally.” Only selected positions contribute to the loss (labels elsewhere are ignored). That sparsity is exactly why ELECTRA’s dense RTD looked attractive.

Strengths and Tradeoffs

Strengths

Unlocks deep bidirectional encoders.
Simple, widely implemented objective.
Strong transfer to classification and spans.

Tradeoffs

~15% positions get gradients.
[MASK] absent at fine-tune time (mitigated by 80/10/10).
Not a full generative language model.

Knowledge Check

Short Answer: What fraction of tokens does classic BERT select for masking? Answer: About 15%.
True/False: Among selected tokens, 80% become [MASK]. Answer: True.
Multiple Choice: MLM loss ignores positions with label: (a) 0, (b) -100 in HF/PyTorch convention, (c) vocab size. Answer: (b).
Short Answer: Why not attend bidirectionally while predicting the next token openly? Answer: The model would see the answer token / future leak.
True/False: Causal LMs and MLM are identical objectives. Answer: False.
Multiple Choice: Dynamic masking: (a) changes masks over training, (b) removes all masks, (c) only masks punctuation. Answer: (a).
Short Answer: Name one mismatch MLM introduces at fine-tuning. Answer: [MASK] token rarely appears downstream.
Short Answer: Which HF pipeline demos MLM? Answer: fill-mask.
Multiple Choice: ELECTRA criticizes MLM mainly for: (a) sparse supervision, (b) using GPUs, (c) Word2Vec. Answer: (a).
True/False: Random and identity replacements (10%/10%) help reduce pretrain/fine-tune gap. Answer: True.

Key Takeaways

MLM is the cloze objective behind BERT-style encoders.
Classic schedule: 15% selected with 80/10/10 replacements.
Loss only on corrupted positions; enables bidirectionality.
Trade sparse supervision for strong NLU representations.
Next: Next Sentence Prediction.

Trainer’s Guide

Hands-on idea: Manually apply 80/10/10 to a 20-token sentence on the board; compute how many positions contribute loss.

Discussion prompt: Would you rather mask 15% or train RTD on 100% of tokens given a fixed FLOP budget?

Recap: MLM teaches bidirectional context via cloze. Finish the module with Next Sentence Prediction.