← Master Index
Vol. 09 Module 9.1 Lecture

Token

NLP Basics

How This Lesson Fits the Module & Volume

With a corpus cleaned in text cleaning, models still cannot read characters as continuous prose the way humans do. They operate on discrete units called tokens.

This lecture defines what a token is—before the next lecture covers tokenization algorithms that produce them. Clarity here prevents confusion later when word tokens, subword pieces, and special tokens ([CLS], <pad>) all appear in the same batch fed to a GRU or transformer.

Learning Objectives

By the end of this lesson, students should be able to:

  • Define a token as the atomic discrete unit of text for an NLP system.
  • Contrast word, subword, character, and special tokens.
  • Explain the relationship between tokens, vocabulary, and integer IDs.
  • Inspect token objects in spaCy and Hugging Face tokenizers.
  • Recognize why token choice affects OOV rate, sequence length, and model cost.
  • Preview how tokens become vectors in Module 9.2 embeddings.
Definition

A token is a discrete textual unit produced by a tokenizer for downstream processing. Depending on the scheme, a token may be a word, a punctuation mark, a subword piece, a character, or a reserved special symbol. Models almost always consume tokens as integer IDs from a fixed vocabulary.

Tokens Are Not Always Words

English speakers often say “word” when they mean “token.” In engineering, the distinction matters:

Token typeExample split of unhappinessTypical use
WordunhappinessClassical NLP, spaCy linguistic pipeline
Subword (BPE/WordPiece)un, ##happiness or unh, appinessTransformers, multilingual models
Characteru n h …Morphology-heavy or noisy text
Special<pad>, <unk>, [SEP]Batching, unknowns, sentence pairs

From Token String to Model Input

1. Text

Cleaned string from the corpus.

2. Token strings

Surface pieces (“cats”, “##ing”).

3. Token IDs

Integers into a vocabulary.

4. Vectors

Embedding lookup (Module 9.2).

Inspecting Tokens in Practice

import spacy from transformers import AutoTokenizer nlp = spacy.load("en_core_web_sm") doc = nlp("GPU prices fell 20% in NYC.") for tok in doc: print(f"{tok.text!r:12} pos={tok.pos_} is_punct={tok.is_punct}") # Subword tokens + IDs for a pretrained model tok = AutoTokenizer.from_pretrained("bert-base-uncased") encoded = tok("GPU prices fell 20% in NYC.") print(tok.convert_ids_to_tokens(encoded["input_ids"])) print(encoded["input_ids"][:8])

Properties Engineers Care About

Vocabulary size

  • Larger vocab → fewer UNKs.
  • Larger embedding matrix.
  • Trade memory vs coverage.

Sequence length

  • Finer tokens → longer sequences.
  • Attention/RNN cost grows.
  • Affects max context windows.

Semantics

  • Word tokens align with linguistics.
  • Subwords handle rare morphology.
  • Special tokens encode structure.

Why Tokens Help

  • Discrete units map cleanly to IDs and embeddings.
  • Enable counting, n-grams, and sequence models.
  • Shared vocabulary makes batching possible.

Pitfalls

  • Confusing spaCy word tokens with BPE pieces.
  • Ignoring special tokens in length budgets.
  • Assuming one true tokenization for all tasks.
Common Misconception

“A token is always a word separated by spaces.” Whitespace splitting fails on punctuation (NYC.), contractions (don't), CJK text (no spaces), and every modern subword tokenizer. Space-separated “words” are only one possible token definition.

Knowledge Check

  1. Short Answer: Define token in NLP engineering terms. Answer: A discrete textual unit produced by a tokenizer for downstream processing (often mapped to an ID).
  2. True/False: Subword pieces are not tokens. Answer: False—they are tokens under a subword scheme.
  3. Multiple Choice: Models typically consume tokens as: (a) PDF pages, (b) integer IDs from a vocabulary, (c) raw UTF-8 only. Answer: (b).
  4. Short Answer: Name three token types. Answer: Word, subword, character (also special tokens).
  5. True/False: <pad> is a special token used for batching variable-length sequences. Answer: True.
  6. Multiple Choice: Finer tokenization usually: (a) shortens sequences, (b) lengthens sequences, (c) removes the vocabulary. Answer: (b).
  7. Short Answer: What comes after token IDs in a neural NLP stack? Answer: Embedding lookup (vectors)—covered in Module 9.2.
  8. True/False: spaCy tokens and BERT WordPiece tokens are guaranteed identical. Answer: False.
  9. Multiple Choice: OOV means: (a) out-of-vocabulary tokens, (b) only very long tokens, (c) optimized vector output. Answer: (a).
  10. Short Answer: Why does token definition affect model cost? Answer: It changes sequence length and vocabulary size, which drive compute and memory.

Key Takeaways

  • A token is the atomic discrete unit your NLP system actually processes.
  • Word, subword, character, and special tokens serve different engineering goals.
  • Tokens map to vocabulary IDs, then to vectors in Module 9.2.
  • Never assume whitespace equals tokenization.
  • Next, Tokenization covers how token sequences are produced.
Trainer’s Guide

Hands-on idea: Have students print spaCy tokens and Hugging Face tokens for the same sentence and list three differences.

Discussion prompt: For a code-mixed Hindi–English chatbot, would you prefer word or subword tokens—and why?

Recap: Tokens are the discrete units models consume; their type is a design choice. Continue with Tokenization.