Next Sentence Prediction — Module 11.2

How This Lesson Fits the Module & Volume

Alongside MLM, original BERT used Next Sentence Prediction to teach relationships between sentence pairs—useful for QA and NLI-style inputs. RoBERTa later dropped NSP; ALBERT replaced it with Sentence Order Prediction. This lecture closes Module 11.2 and bridges to Module 11.3’s generative GPT-1.

Learning Objectives

By the end of this lesson, students should be able to:

Define NSP and how positive/negative pairs are constructed.
Describe how [CLS] is used for the binary NSP head.
Explain critiques of NSP and why RoBERTa removed it.
Contrast NSP with ALBERT’s SOP.
Relate sentence-pair packing to downstream QA/NLI formats.
Place encoder pair modeling vs. GPT-style single-stream generation.

Definition

Next Sentence Prediction is a binary classification pretraining task: given sentences A and B, predict whether B is the true consecutive next sentence after A (IsNext) or a random sentence from the corpus (NotNext).

How NSP Is Built

Sample A

Take a sentence (or span) A.

50% IsNext

B = actual following sentence.

50% NotNext

B = random sentence.

Classify

[CLS] → binary logits.

Inputs look like [CLS] A [SEP] B [SEP] with segment IDs marking A vs. B. The NSP head is trained jointly with MLM.

Objective	Positive	Negative	What it tests
NSP (BERT)	True next sentence	Random sentence	Often topic + discourse
SOP (ALBERT)	Correct order	Swapped order	Fine-grained coherence
None (RoBERTa)	—	—	Full-sentence MLM only

Why NSP Fell Out of Favor

Random negatives are frequently from different topics, so the model can succeed via shallow topic discrimination rather than true discourse relations. Longer training on packed documents without NSP matched or beat BERT on many benchmarks—hence RoBERTa’s removal.

Keeps NSP useful for

Teaching historical BERT
Some pair-format tasks
Understanding segment embeddings

Prefer alternatives

SOP for order sensitivity
Document packing + MLM
Task-specific pair fine-tuning

Vs. GPT world

GPT predicts tokens, not IsNext
Dialogue is left-to-right
Next module: GPT-1

Conceptual Code

import torch
from torch import nn

class NspHead(nn.Module):
    def __init__(self, hidden=768):
        super().__init__()
        self.fc = nn.Linear(hidden, 2)  # IsNext / NotNext

    def forward(self, cls_hidden):
        return self.fc(cls_hidden)

head = NspHead()
cls_vec = torch.randn(4, 768)  # batch of [CLS] states
print(head(cls_vec).shape)     # torch.Size([4, 2])

Common Misconception

“NSP is required for any sentence-pair BERT fine-tune.” Downstream NLI/QA fine-tuning works well without NSP pretraining; the pair format matters more than the original NSP head.

Strengths and Tradeoffs

Strengths

Historically helped pair-aware representations.
Simple binary auxiliary loss.
Teaches segment embeddings clearly.

Tradeoffs

Often too easy via topic cues.
RoBERTa-style recipes omit it successfully.
Less central than MLM for encoder quality.

Knowledge Check

Short Answer: What does NSP predict? Answer: Whether B is the true next sentence after A.
True/False: Classic NSP uses roughly 50% random negatives. Answer: True.
Multiple Choice: Which vector usually feeds the NSP classifier? (a) [CLS], (b) last token only, (c) pixel embedding. Answer: (a).
Short Answer: Why can NSP be too easy? Answer: Random sentences often differ in topic.
True/False: RoBERTa keeps NSP unchanged from BERT. Answer: False—it removes NSP.
Multiple Choice: ALBERT’s SOP negatives are: (a) swapped order, (b) images, (c) empty strings. Answer: (a).
Short Answer: Show the token pattern for a pair. Answer: [CLS] A [SEP] B [SEP].
Short Answer: Is NSP the main reason BERT works? Answer: No—MLM is the primary driver; NSP is auxiliary/historical.
Multiple Choice: Module 11.3 shifts focus to: (a) GPT generative models, (b) only CNNs, (c) only k-NN. Answer: (a).
True/False: Segment embeddings help the model know which tokens belong to A vs. B. Answer: True.

Key Takeaways

NSP is BERT’s original sentence-pair auxiliary task.
IsNext vs. NotNext via [CLS]; often shallow topic signal.
RoBERTa drops NSP; ALBERT prefers SOP.
MLM remains the central encoder pretraining idea.
Next module: GPT-1 and the decoder-only family.

Trainer’s Guide

Hands-on idea: Build 4 toy pairs (2 IsNext, 2 NotNext) and discuss which cues a bag-of-words topic model could already use.

Discussion prompt: If you redesign BERT today, do you keep NSP, use SOP, or drop pair objectives entirely?

Recap: NSP taught pair relations—then the field moved on. Continue to Module 11.3 with GPT-1.