BERT — Module 11.2

How This Lesson Fits the Module & Volume

Volume 10 built the Transformer: self-attention, encoders, and bidirectional stacks. Module 11.1 covered language-model mechanics—tokens, embeddings, and inference.

BERT opens Module 11.2: the encoder-only family that redefined NLU via masked language modeling and next-sentence prediction. Later lectures refine BERT into RoBERTa, ALBERT, DistilBERT, ELECTRA, and Sentence-BERT.

Learning Objectives

By the end of this lesson, students should be able to:

Define BERT as a bidirectional encoder Transformer pretrained with MLM (+ original NSP).
Explain why bidirectionality helps NLU more than left-to-right LMs for classification and span tasks.
Describe WordPiece inputs, special tokens ([CLS], [SEP], [MASK]), and segment embeddings.
Contrast pretrain vs. fine-tune for GLUE-style tasks.
Load a Hugging Face BERT checkpoint and run a classification forward pass.
Place BERT relative to GPT-style decoder-only models (Module 11.3).

Definition

BERT (Bidirectional Encoder Representations from Transformers, Devlin et al., 2018) is an encoder-only Transformer pretrained so every token can attend to its full left and right context. The original objectives were Masked Language Modeling and Next Sentence Prediction; the resulting contextual vectors transfer strongly to downstream NLU after light fine-tuning.

Why Bidirectional Context Matters

Left-to-right models (classic GPT) predict the next token from a prefix. That is ideal for generation, but for sentence classification, NER, or extractive QA you usually need both sides of a word. BERT’s encoder stack uses full self-attention (no causal mask), so the representation of “bank” in “river bank” vs. “bank account” can use the whole sentence.

Encoder-only (BERT)

Bidirectional attention
MLM / NLU heads
Strong at understanding

Decoder-only (GPT)

Causal attention
Next-token prediction
Strong at generation

Encoder–Decoder

Bidirectional encode + causal decode
Seq2seq / T5 style
Translation, summarization

Input Format

BERT tokenizes with WordPiece. Sequences start with [CLS] (often pooled for classification) and separate sentence pairs with [SEP]. Token, segment (A/B), and position embeddings sum before the first layer.

Variant	Layers	Hidden	Heads	~Params
BERT-Base	12	768	12	110M
BERT-Large	24	1024	16	340M

Pretraining Objectives (Original)

1. MLM

Mask ~15% of tokens; predict originals from context.

2. NSP

Is sentence B the true next sentence after A?

3. Fine-tune

Replace head; train on labeled task data.

Later work (especially RoBERTa) showed NSP is often unnecessary; the family still starts from BERT’s design. Deep dives: Masked Language Modeling and Next Sentence Prediction.

Fine-Tuning Pattern

For classification, take the [CLS] hidden state (or mean pooling), add a linear layer, and train with cross-entropy. For span QA, predict start/end logits over token positions. Only a few epochs on modest labeled data are typical because pretraining already encoded language structure.

Hugging Face + PyTorch

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

name = "bert-base-uncased"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name, num_labels=2)

batch = tok(["This movie was excellent.", "Terrible plot."],
            padding=True, truncation=True, return_tensors="pt")
with torch.no_grad():
    logits = model(**batch).logits
print(logits.argmax(-1))  # class ids

Strengths and Tradeoffs

Strengths

State-of-the-art NLU transfer circa 2018–2020.
Simple fine-tune recipe across many tasks.
Huge ecosystem of checkpoints and tutorials.

Tradeoffs

Not designed for open-ended generation.
Fixed max length (originally 512 WordPiece tokens).
MLM under-trains masked positions vs. every-token LM loss.

Common Misconception

“BERT is a language model that writes text like ChatGPT.” BERT is primarily an understanding encoder. You can fill masks, but fluent long-form generation is the GPT / decoder-only story in Module 11.3.

Knowledge Check

Short Answer: What does BERT stand for? Answer: Bidirectional Encoder Representations from Transformers.
True/False: BERT uses a causal (left-to-right only) attention mask during pretraining. Answer: False—attention is bidirectional.
Multiple Choice: The original BERT pretraining pair was: (a) RLHF + DPO, (b) MLM + NSP, (c) CTC + CTC. Answer: (b).
Short Answer: Which special token is commonly used as a sentence classification vector? Answer: [CLS].
True/False: BERT-Base has 12 layers and hidden size 768. Answer: True.
Multiple Choice: BERT’s tokenizer family is: (a) WordPiece, (b) only byte-level BPE like GPT-2, (c) characters only. Answer: (a).
Short Answer: Name one reason MLM enables bidirectionality without “seeing the answer.” Answer: Masked tokens are hidden; the model predicts them from surrounding context.
Short Answer: For extractive QA, what do start/end heads predict? Answer: Logits over token positions for answer span boundaries.
Multiple Choice: Compared with GPT-style models, BERT is stronger at: (a) open-ended story writing by default, (b) NLU classification / span tasks, (c) diffusion image synthesis. Answer: (b).
True/False: Fine-tuning usually trains from scratch with random weights. Answer: False—you start from pretrained weights.

Key Takeaways

BERT is an encoder-only bidirectional Transformer for NLU transfer.
Original pretraining: MLM + NSP; fine-tune with task heads.
Inputs combine WordPiece tokens with [CLS]/[SEP] and segment IDs.
Hugging Face makes loading and fine-tuning BERT routine in PyTorch.
Next: RoBERTa removes NSP and scales training carefully.

Trainer’s Guide

Hands-on idea: Fine-tune bert-base-uncased on a tiny binary sentiment set; compare frozen encoder vs. full fine-tune accuracy.

Discussion prompt: When would you still pick BERT over a small instruction-tuned LLM for production classification?

Recap: BERT made bidirectional Transformers the default for understanding tasks. Continue with RoBERTa.