Temperature

How This Lesson Fits the Module & Volume

Sampling needs a distribution. Temperature rescales logits before softmax, controlling how peaked or flat that distribution is—without changing which token has the highest raw score.

It is the first decoding hyperparameter most products expose. Later you combine it with top-k / top-p for safer creativity.

Learning Objectives

By the end of this lesson, students should be able to:

Define temperature as dividing logits by T before softmax.
Predict behavior as T → 0, T = 1, and T > 1.
Implement temperature sampling in PyTorch.
Explain why T does not reorder argmax at fixed logits.
Choose sensible T ranges for factual vs creative tasks.
Combine temperature with truncation strategies conceptually.

Definition

Temperature T > 0 reshapes the next-token distribution via

p_i = softmax(z_i / T) = exp(z_i / T) / Σ_j exp(z_j / T).

T = 1 leaves logits unchanged. T < 1 sharpens (more confident / greedy-like). T > 1 flattens (more random / diverse). As T → 0, sampling approaches greedy argmax.

Effect of T on the Distribution

T → 0

Near one-hot on max

T = 1

Model’s native probs

T > 1

Flatter, more entropy

T → ∞

Nearly uniform

Setting	Distribution	Typical use
T ≈ 0–0.3	Very peaked	Code, short facts, formats
T ≈ 0.7–1.0	Balanced	General chat defaults
T ≈ 1.2–1.5	Higher entropy	Brainstorming, fiction
T ≫ 2	Often degraded	Usually avoided

Temperature vs Truncation

Rescales all logits.
Does not remove tokens.
Tail still has some mass if T high.

Top-k / Top-p

Hard-zero low tokens.
Renormalize remaining mass.
Blocks rare disasters.

Together

Apply T, then truncate, then sample.
Common API order in libraries.
Tune both for product quality.

Code: Temperature Sampling

import torch
import torch.nn.functional as F

def sample_with_temperature(logits, temperature=1.0):
    # logits: (B, V)
    if temperature <= 0:
        return logits.argmax(dim=-1, keepdim=True)
    scaled = logits / temperature
    probs = F.softmax(scaled, dim=-1)
    return torch.multinomial(probs, num_samples=1)

logits = torch.tensor([[4.0, 2.0, 1.0, 0.0]])
torch.manual_seed(1)
print("T=0.2", sample_with_temperature(logits, 0.2).item())
torch.manual_seed(1)
print("T=1.5", sample_with_temperature(logits, 1.5).item())

# Entropy rises with T
for T in [0.5, 1.0, 2.0]:
    p = F.softmax(logits / T, dim=-1)
    ent = -(p * (p + 1e-12).log()).sum()
    print(f"T={T}: entropy={ent.item():.3f}")

Strengths and Tradeoffs

Strengths

One scalar controls creativity vs focus.
Cheap: a divide before softmax.
Intuitive product knob.

Tradeoffs

High T still allows bad tail tokens.
Does not fix factual errors by itself.
T ≤ 0 must be special-cased (greedy).

Common Misconception

“Temperature changes which token is most likely.” For a fixed logit vector, dividing by T > 0 preserves the argmax. Temperature changes how much probability mass the top tokens get relative to the rest—not the ranking of the scores.

Related module pages: Sampling, Logits, Softmax, Top-K, Top-P.

Knowledge Check

Short Answer: How is temperature applied to logits? Answer: Divide logits by T before softmax.
True/False: T = 1 leaves the distribution unchanged vs raw logits. Answer: True.
Multiple Choice: T < 1 makes the distribution: (a) sharper, (b) uniform, (c) undefined. Answer: (a).
Short Answer: What happens as T → 0? Answer: Sampling approaches greedy argmax.
True/False: Temperature reorders which logit is largest. Answer: False (for T > 0).
Multiple Choice: High T increases: (a) entropy, (b) model parameters, (c) vocab size. Answer: (a).
Short Answer: Why is high T alone risky? Answer: More mass on low-quality tail tokens.
True/False: Temperature replaces the need for top-p. Answer: False—they solve different problems and are often combined.
Multiple Choice: For strict JSON generation, prefer: (a) high T, (b) low T, (c) T = ∞. Answer: (b).
Short Answer: Which truncation keeps a fixed number of top tokens? Answer: Top-k.

Key Takeaways

Temperature divides logits before softmax to control sharpness.
Low T ≈ greedy; high T ≈ more random.
Argmax ranking is unchanged for T > 0.
Often combined with top-k / top-p for quality.
Next: Top-K.

Trainer’s Guide

Hands-on idea: Plot or print probability vectors for the same logits at T ∈ {0.2, 1, 2} and have students predict sample histograms before running multinomial.

Discussion prompt: Should temperature be user-facing, or an internal A/B-tested default per feature?

Recap: Temperature rescales logits to control how peaked sampling is. Continue with Top-K.