← Master Index
Vol. 11 Module 11.1 Lecture

Tokens

Language Model Concepts

How This Lesson Fits the Module & Volume

The vocabulary lists token types. A token is an occurrence of one of those types in a sequence—the atomic item that fills the context window, gets an embedding, and is scored by next-token prediction.

Vol. 09’s token lecture introduced the linguistic idea. Here we focus on LM practice: token IDs as tensors, whitespace quirks, multibyte scripts, and why billing / latency are quoted in tokens.

Learning Objectives

By the end of this lesson, students should be able to:

  • Distinguish token type (vocab entry) from token instance (sequence position).
  • Explain why tokens are not the same as words, characters, or bytes (though they can be).
  • Read and manipulate input_ids tensors in PyTorch / Hugging Face.
  • Estimate how tokenization affects sequence length for English vs code vs CJK text.
  • Describe special tokens as ordinary IDs with reserved roles.
  • Prepare for the next lecture: how a tokenizer maps text ↔ token IDs.
Definition

A token is the atomic discrete unit of text representation in an LM pipeline: a vocabulary ID at a sequence position. Models never see raw Unicode strings inside the Transformer—only sequences of these IDs (then vectors).

Tokens vs Words vs Characters

UnitExample split of “unhappiness!”Typical use
Charactersu n h a p p i n e s s !Char-RNNs; very long seqs
Wordsunhappiness !Classical NLP; OOV risk
Subword tokensun happiness ! (schema-dependent)Modern LMs
BytesUTF-8 byte IDsByte-level BPE fallbacks

In the Model

  • Shape (batch, seq_len) of ints.
  • Each int ∈ [0, |V|).
  • Position index matters for PE / RoPE.

In Products

  • API pricing per token.
  • Rate limits & context quotas.
  • Streaming = token chunks.

Common Surprises

  • Leading spaces are often tokens.
  • Digits may split (“2026” → pieces).
  • Code and CJK can be token-heavy.

Code: IDs Are What the Model Sees

import torch from transformers import AutoTokenizer, AutoModelForCausalLM tok = AutoTokenizer.from_pretrained("gpt2") model = AutoModelForCausalLM.from_pretrained("gpt2").eval() text = "Tokens are not words." enc = tok(text, return_tensors="pt") ids = enc["input_ids"] # (1, T) print(ids) print(tok.convert_ids_to_tokens(ids[0])) with torch.no_grad(): out = model(input_ids=ids) print("logits shape:", tuple(out.logits.shape)) # (1, T, vocab_size) # Detokenize a continuation id next_id = int(out.logits[0, -1].argmax()) print("greedy next:", repr(tok.decode([next_id])))

Type vs Instance

The vocabulary entry for " the" is one type. In the sentence “the cat and the dog,” that type may appear twice as two instances at different positions—same ID, different contextual hidden states after the stack. Beginners sometimes think repeated words share one vector inside the Transformer; only the lookup embedding is shared before context mixes.

Why Subword Tokens Win

  • Open-vocabulary behavior.
  • Stable |V| for the LM head.
  • Shares pieces across word forms.

Costs

  • Human-unfriendly splits.
  • Token tax on some languages/code.
  • Debugging needs detokenization care.
Common Misconception

“One token equals one word.” In LM stacks, a word may become several tokens, and one token may be a fragment, a space-prefixed word, punctuation, or a control marker. Always count with the model’s tokenizer. Product copy that says “words” when it means tokens misleads users about context limits.

Knowledge Check

  1. Short Answer: What tensor does a causal LM consume as text input? Answer: Integer token IDs, shape (batch, seq_len).
  2. True/False: Tokens are always whole English words. Answer: False.
  3. Multiple Choice: Vocabulary type vs token instance: (a) same meaning, (b) type is the vocab entry; instance is a position in a sequence, (c) instance is the embedding dim. Answer: (b).
  4. Short Answer: Why might Chinese text use more tokens than an English translation? Answer: Tokenizers often allocate fewer merges to CJK, so characters split into more pieces (any clear wording).
  5. True/False: Two occurrences of the same token ID share the same contextual hidden state after deep layers. Answer: False—embeddings start the same; context differentiates them.
  6. Multiple Choice: API billing commonly meters: (a) UTF-8 bytes only, (b) tokens, (c) GPU FLOPs billed to users directly. Answer: (b).
  7. Short Answer: What does convert_ids_to_tokens show that decode might hide? Answer: The raw piece strings (including Ġ/spacemarkers) before string cleanup.
  8. Short Answer: Name one non-word token kind. Answer: Punctuation, leading-space piece, EOS, byte fallback, etc.
  9. Multiple Choice: Logits shape (1, T, V) means: (a) V sequences, (b) per-position scores over vocab size V, (c) T vocabularies. Answer: (b).
  10. True/False: Special tokens are still integer IDs in the same ID space. Answer: True.

Key Takeaways

  • Tokens are the discrete units—IDs in a sequence—that LMs actually process.
  • They need not align with words; subwords dominate modern systems.
  • Same ID can appear many times; contextual states still diverge.
  • Length, cost, and context limits are token-based metrics.
  • Next: Tokenizer—the map between text and token IDs.
Trainer’s Guide

Hands-on idea: Tokenize the same string with GPT-2 and BERT WordPiece; compare piece lists side by side.

Discussion prompt: Should a documentation site explain limits in tokens or approximate words? What goes wrong either way?

Recap: Tokens are vocabulary IDs in sequence positions—the true currency of LM context and compute. Continue with Tokenizer.