Top-P — Module 11.1

How This Lesson Fits the Module & Volume

Top-k always keeps k tokens. When the model is highly confident, that may be too many; when it is uncertain, too few. Top-p (nucleus) sampling keeps the smallest set of top tokens whose cumulative probability is at least p.

Nucleus sampling (Holtzman et al.) is the adaptive truncation used widely in chat and creative generation, usually combined with mild temperature.

Learning Objectives

By the end of this lesson, students should be able to:

Define nucleus / top-p sampling via cumulative probability.
Implement top-p filtering on sorted probabilities in PyTorch.
Explain why the candidate set size adapts per step.
Compare top-p with top-k on peaked vs flat distributions.
Choose typical p values (e.g. 0.9–0.95) for practice.
Combine temperature, top-k, and top-p in one pipeline.

Definition

Top-p (nucleus) sampling sorts tokens by descending probability, includes tokens until their cumulative mass ≥ p, discards the rest (logits → −∞), renormalizes, and samples. The nucleus is the smallest prefix of the ranked list that covers probability p.

Building the Nucleus

Softmax

Get probs

Sort

Descending p

Cumsum

Find cutoff

Sample

In the nucleus

p	Nucleus size trend	Notes
p = 0.0–ish edge	Often 1 token	Near-greedy if only top mass kept carefully
p = 0.9	Moderate	Common chat default
p = 0.95	Larger	More diversity
p = 1.0	Full vocab	No truncation

Adaptive vs Fixed Truncation

Peaked step

Top token has mass 0.92.
Top-p=0.9 may keep ~1–2 tokens.
Top-k=40 would still allow 40.

Flat step

Mass spread over many tokens.
Top-p expands the nucleus.
Top-k may cut needed mass.

API combo

Temperature first.
Optional top-k AND top-p.
Intersection of filters.

Code: Top-P Filter + Sample

import torch
import torch.nn.functional as F

def top_p_logits(logits, p=0.9):
    # logits: (B, V)
    sorted_logits, sorted_idx = torch.sort(logits, descending=True)
    probs = F.softmax(sorted_logits, dim=-1)
    cumprobs = torch.cumsum(probs, dim=-1)

    # mask tokens that push cumprob strictly beyond p (keep first that crosses)
    mask = cumprobs - probs > p
    sorted_logits = sorted_logits.masked_fill(mask, float("-inf"))

    # scatter back to original vocab order
    out = torch.full_like(logits, float("-inf"))
    out.scatter_(1, sorted_idx, sorted_logits)
    return out

def sample_top_p(logits, p=0.9, temperature=1.0):
    if temperature > 0:
        logits = logits / temperature
    filtered = top_p_logits(logits, p)
    probs = F.softmax(filtered, dim=-1)
    return torch.multinomial(probs, num_samples=1)

logits = torch.randn(1, 500)
torch.manual_seed(0)
print(sample_top_p(logits, p=0.92, temperature=0.9))

Strengths and Tradeoffs

Strengths

Adapts candidate set to confidence.
Strong quality/diversity tradeoff in practice.
Standard in modern decoding stacks.

Tradeoffs

Slightly more logic than top-k (sort + cumsum).
Off-by-one bugs around the cumprob threshold are common.
Still stochastic; not a search for global max likelihood.

Common Misconception

“Top-p = 0.9 means each token needs probability ≥ 0.9.” No: p is a cumulative threshold. You keep the highest-ranked tokens until their probabilities add up to at least 0.9. Individual tokens inside the nucleus can be far smaller than 0.9.

Related module pages: Top-K, Temperature, Sampling, Beam Search, Inference.

Knowledge Check

Short Answer: What is another name for top-p sampling? Answer: Nucleus sampling.
True/False: Top-p always keeps a fixed number of tokens. Answer: False—the set size adapts.
Multiple Choice: p refers to: (a) cumulative probability mass, (b) learning rate, (c) pad id. Answer: (a).
Short Answer: On a peaked distribution, does the nucleus tend to shrink or grow? Answer: Shrink.
True/False: Top-p = 1.0 disables truncation. Answer: True.
Multiple Choice: Implementation usually needs: (a) sort + cumsum, (b) only BatchNorm, (c) k-means. Answer: (a).
Short Answer: Why is “each token ≥ 0.9” wrong? Answer: p is cumulative mass, not a per-token minimum.
True/False: Temperature is often applied before top-p. Answer: True.
Multiple Choice: Top-p is primarily: (a) sampling with truncation, (b) training loss, (c) tokenizer BPE merge. Answer: (a).
Short Answer: Which lecture covers keeping multiple hypotheses instead of sampling one? Answer: Beam Search.

Key Takeaways

Top-p keeps the smallest top set with cumulative mass ≥ p.
The nucleus adapts to model confidence each step.
p is cumulative—not a per-token threshold.
Often combined with temperature and sometimes top-k.
Next: Beam Search.

Trainer’s Guide

Hands-on idea: For one logit row, print nucleus sizes at p ∈ {0.5, 0.9, 0.95} and compare to top-k=40 membership.

Discussion prompt: If an API exposes both top-k and top-p, how should product docs explain their interaction?

Recap: Top-p (nucleus) sampling truncates by cumulative probability for adaptive diversity. Continue with Beam Search.