Logits — Module 11.1

How This Lesson Fits the Module & Volume

The hidden state is still a d_model-dimensional vector. To choose a token from a vocabulary of size V, the model projects that vector into V scores called logits.

Every decoding knob later—temperature, top-k, top-p, beam search—operates on logits (or on probabilities derived from them). Master logits before you tune generation.

Learning Objectives

By the end of this lesson, students should be able to:

Define logits as unnormalized vocabulary scores.
Relate the LM head linear layer to shape (B, V) or (B, T, V).
Explain why logits are not probabilities until softmax.
Connect cross-entropy loss to logits during training.
Identify where decoding strategies edit or rescale logits.
Implement a minimal LM head projection in PyTorch.

Definition

Logits are the raw, unbounded scores the model assigns to each vocabulary token before converting them into a probability distribution. If h ∈ R^{d_model} is the relevant hidden state and W ∈ R^{V × d_model} is the LM head weight, then logits z = W h (plus optional bias). Softmax turns z into probabilities.

The Scoring Pipeline

Hidden h

(B, d) or (B, T, d)

LM head

Linear map to V

Logits z

Unnormalized scores

Softmax

Valid probabilities

Property	Logits	Probabilities
Range	(−∞, +∞)	[0, 1]
Sum constraint	None	Must sum to 1
Training loss	Cross-entropy takes logits	Usually not stored explicitly
Decoding edits	Mask, scale (temperature), truncate	Renormalize after edits

Training vs Inference Use

Training

F.cross_entropy(logits, targets)
Softmax is inside the loss numerically stably.
All positions may produce logits (B, T, V).

Greedy decode

token = logits.argmax(-1)
No sampling noise.
Deterministic given the model.

Stochastic decode

Rescale / filter logits first.
Then softmax + sample.
See Sampling & Temperature lectures.

Code: Logits from Hidden State

import torch
import torch.nn as nn
import torch.nn.functional as F

B, d, V = 2, 64, 5000
h = torch.randn(B, d)                 # final hidden states
lm_head = nn.Linear(d, V, bias=False)

logits = lm_head(h)                   # (B, V)
print(logits.shape, logits.min().item(), logits.max().item())

# Training: loss on logits (softmax inside)
targets = torch.randint(0, V, (B,))
loss = F.cross_entropy(logits, targets)

# Greedy next token
next_id = logits.argmax(dim=-1)       # (B,)
print(next_id)

Strengths and Tradeoffs

Why work in logit space

Numerically friendly for loss and masking (−∞).
Temperature is a simple divide on logits.
Top-k / top-p filters are easy before softmax.

Pitfalls

Treating logits as probabilities misleads thresholds.
Weight tying with embeddings changes how you interpret W.
Huge V makes storing full (B, T, V) expensive.

Common Misconception

“The largest logit is already a probability.” A logit of 12.4 is not “12.4%” or “probability 12.4.” Only after softmax (or an equivalent normalization) do scores become probabilities that sum to one. Always convert before interpreting likelihoods.

Related module pages: Hidden State, Softmax, Vocabulary, Probability Distribution, Sampling.

Knowledge Check

Short Answer: What shape do next-token logits usually have? Answer: (B, V) for one step, or (B, T, V) for a full sequence.
True/False: Logits are guaranteed to lie in [0, 1]. Answer: False.
Multiple Choice: Cross-entropy in PyTorch typically expects: (a) probabilities, (b) logits, (c) token strings. Answer: (b).
Short Answer: What linear layer maps h → logits? Answer: The LM head (output projection).
True/False: Temperature decoding divides logits before softmax. Answer: True.
Multiple Choice: Greedy decoding picks: (a) argmax of logits, (b) argmin of logits, (c) a random pad. Answer: (a).
Short Answer: Why set forbidden tokens to −∞ in logit space? Answer: Softmax maps −∞ to probability ~0.
True/False: A logit of 5 means probability 0.5. Answer: False.
Multiple Choice: V in (B, V) is: (a) vocabulary size, (b) batch size, (c) depth. Answer: (a).
Short Answer: Which lecture turns logits into a distribution? Answer: Softmax.

Key Takeaways

Logits are unnormalized vocab scores from the LM head.
Training losses and decoding filters operate on logits.
They are not probabilities until softmax (or equivalent).
Greedy decode = argmax over logits.
Next: Softmax.

Trainer’s Guide

Hands-on idea: Print top-5 logits and top-5 probabilities for the same vector; ask students which list they would trust for “% chance.”

Discussion prompt: When is weight tying (sharing embed and LM head) helpful, and when might you keep them separate?

Recap: Logits score every vocabulary token from a hidden state; decoding and loss both start here. Continue with Softmax.