Perplexity — Module 19.1

How This Lesson Fits the Module & Volume

Vol. 19 opened with classification metrics (accuracy, precision, recall, F1, ROC) and n-gram overlap (BLEU, ROUGE). Those score tasks. Perplexity scores the language model itself: how surprised is it by held-out text under its own next-token distribution?

It sits between Vol. 11 next-token prediction / probability distributions and the operational metrics that follow—latency and token usage. Low perplexity is not “the chatbot is truthful.” It is “the LM assigns high probability to this corpus.”

Learning Objectives

By the end of this lesson, students should be able to:

Define perplexity as exponentiated mean cross-entropy over tokens.
Compute PPL from per-token log-probabilities on a held-out set.
Explain why PPL is tokenizer-dependent and incomparable across vocabularies.
Contrast perplexity with task metrics (accuracy, BLEU, human eval).
Name fairer alternatives (bits-per-byte / bits-per-character) when tokenizers differ.
State when PPL is useful (pretraining, domain fit) and when it is misleading (chat quality).

Definition

Perplexity (PPL) of a language model on a token sequence \(x_1,\ldots,x_N\) is \(\mathrm{PPL}=\exp(\mathrm{CE})\), where \(\mathrm{CE}=-\frac{1}{N}\sum_{i=1}^{N}\log p(x_i\mid x_{<i})\) is the average cross-entropy (nats if \(\log\) is natural). Equivalently, \(\mathrm{PPL}=\exp\bigl(-\frac{1}{N}\sum_i\log p(x_i\mid x_{<i})\bigr)=\bigl(\prod_i p(x_i\mid x_{<i})\bigr)^{-1/N}\). Lower is better: the model is less “perplexed” by the text. \(N\) is almost always a token count under that model’s tokenizer.

From Cross-Entropy to Perplexity

A causal LM emits a distribution over the vocabulary at each step. Cross-entropy on the true next token is the standard training loss. Perplexity is that loss, exponentiated, so a CE of \(0\) nats \(\to\) PPL \(1\) (perfect), CE of \(\ln 2\approx 0.69\) \(\to\) PPL \(2\), and so on. Intuition: PPL is the effective branching factor—an average “how many equally likely tokens” the model is choosing among.

Mean CE (nats)	Perplexity	Reading
0	1	Deterministic match to the corpus
\(\ln 2 \approx 0.69\)	2	Like a fair coin at each step
\(\ln 10 \approx 2.30\)	10	Effective 10-way choice
\(\ln V\)	\(V\)	Uniform over vocab size \(V\)

Computing PPL on Held-Out Text

Always evaluate on data the model did not train on. Teacher-force the true prefix (do not sample); sum log-probs of the true tokens. Sliding windows or packing matter for long documents—report how you chunked.

# Toy PPL from per-token log-probs (natural log). Replace with model.forward().
import math

def perplexity(log_probs):
    # log_probs: list of ln p(true_token | prefix) for each position
    n = len(log_probs)
    if n == 0:
        raise ValueError("empty sequence")
    ce = -sum(log_probs) / n
    return math.exp(ce), ce

# Example: three tokens with p = 0.4, 0.25, 0.5
lp = [math.log(0.4), math.log(0.25), math.log(0.5)]
ppl, ce = perplexity(lp)
print(f"CE={ce:.3f} nats  PPL={ppl:.2f}")
# CE=1.050  PPL=2.86

# Hugging Face-style sketch (causal LM):
# loss = model(input_ids, labels=input_ids).loss  # mean CE in nats
# ppl = torch.exp(loss)

# Bits-per-byte (tokenizer-agnostic-ish):
# bpb = (-sum(log_probs) / math.log(2)) / num_bytes

Tokenizer Dependence

PPL is an average per token. A tokenizer that splits the same sentence into more tokens changes \(N\) and the per-step probabilities. Two models with different BPE / WordPiece / SentencePiece vocabs can report wildly different PPL on identical bytes. You must not rank Model A vs Model B by PPL unless they share the same tokenizer (or you convert to bits-per-byte / bits-per-character).

Same tokenizer

PPL comparisons are meaningful.
Typical for ablation of one architecture family.
Still need the same eval corpus and packing.

Different tokenizer

PPL is not comparable.
Use bits/byte or a shared byte-level eval.
Word-level PPL is an old compromise, still lossy.

API / closed models

You usually cannot read token log-probs.
PPL is a pretraining / open-weight tool.
Product eval → tasks, not PPL.

What PPL Does and Does Not Measure

Use PPL for	Do not use PPL for
Pretraining progress on a fixed tokenizer + corpus	Chat helpfulness or style
Domain fit (news LM vs code LM on code dumps)	Factuality / hallucination
Detecting train/eval leakage (suspiciously tiny PPL)	Ranking APIs on a leaderboard
Comparing checkpoints of one training run	Substitution for human evaluation

Strengths

Cheap, automatic, differentiable cousin of the loss.
No labels beyond raw text.
Sensitive to domain mismatch.

Limits

Tokenizer + length + packing confounders.
A fluent liar can have excellent PPL.
Instruction-tuned chat models are not scored fairly by wiki PPL alone.

Related Lectures

Lecture	Role
Language model / NTP	What PPL is averaging
Tokenizer / tiktoken	Why \(N\) and PPL move together
BLEU / ROUGE	Task overlap metrics, not LM likelihood
Latency	Next: user-visible time, not likelihood
Benchmarks	MMLU-style tasks vs intrinsic PPL

Common Misconception

“Lower perplexity means a better product model.” PPL measures fit to a text distribution, not truth, safety, or instruction following. Second: comparing GPT-style PPL to Llama-style PPL across different tokenizers. Third: reporting PPL on the training set (memorization looks like genius). Fourth: exponentiating a loss that was already averaged in bits without converting bases. Fifth: thinking sampling temperature changes PPL—PPL uses the model’s probabilities on gold tokens, not sampled ones.

Knowledge Check

Short Answer: Write PPL in terms of mean cross-entropy CE. Answer: PPL = exp(CE) (with CE in nats).
True/False: Higher perplexity always means a better language model. Answer: False—lower is better.
Multiple Choice: PPL is primarily: (a) an LM intrinsic metric, (b) a RAG faithfulness score, (c) TTFT. Answer: (a).
Short Answer: Why can’t you compare PPL across different tokenizers? Answer: Token count \(N\) and per-token probs change with the vocabulary/segmentation.
True/False: A model can have low wiki PPL and still hallucinate in chat. Answer: True.
Multiple Choice: A tokenizer-fairer report is: (a) bits-per-byte, (b) BLEU-4 only, (c) p95 latency. Answer: (a).
Short Answer: What sequence do you score—sampled or gold tokens? Answer: Gold / held-out true tokens (teacher forcing).
True/False: Closed APIs usually expose full token log-probs for PPL. Answer: False—often unavailable.
Multiple Choice: Next lecture: (a) Latency, (b) PCA, (c) DreamBooth. Answer: (a).
Short Answer: What does suspiciously tiny eval PPL sometimes indicate? Answer: Train/eval leakage or memorization.

Key Takeaways

Perplexity \(=\exp(\text{mean token cross-entropy})\); lower \(\Rightarrow\) better LM fit.
It is tokenizer-dependent; use bits/byte (or a shared tokenizer) to compare families.
PPL \(\neq\) accuracy, BLEU, groundedness, or UX.
Best for pretraining, domain fit, and checkpoint tracking—not product leaderboards.
Next: Latency.

Trainer’s Guide

Lab: Compute PPL of a small causal LM on two held-out files (news vs code). Then retokenize the same bytes with a second vocab and show PPL jumps while bits/byte stays closer. Plot CE vs PPL for a few CE values.

Whiteboard: \(\mathrm{CE}\xrightarrow{\exp}\mathrm{PPL}\). Arrow “different BPE” breaking comparability. Arrow to Latency: users feel time, not nats.

Recap: Perplexity exponentiates cross-entropy to score how well an LM predicts held-out tokens—powerful, tokenizer-bound, and not a substitute for task or human eval. Measure user-visible time next with Latency.