Module 9.1 turned raw text into linguistic units: tokenization, cleaning, stemming, lemmatization, POS tags, and dependency structure. Those steps produce symbols—strings and token IDs—but neural networks and classical ML models need numbers.
One-hot encoding is the bridge into Module 9.2. It is the simplest numeric representation of discrete vocabulary items: each word becomes a sparse binary vector. Everything that follows—TF-IDF, bag of words, Word2Vec, GloVe, FastText, and the trainable embedding layer—exists to overcome one-hot’s sparsity and lack of meaning.
Learning Objectives
By the end of this lesson, students should be able to:
- Explain why tokenized text must be converted to numeric vectors before modeling.
- Construct one-hot vectors for a small vocabulary and encode a short sentence.
- State the dimensionality and sparsity properties of one-hot word vectors.
- Implement one-hot encoding with sklearn and a manual PyTorch lookup.
- Identify the limitations that motivate dense embeddings later in this module.
- Place one-hot encoding on the evolutionary path from discrete IDs to learned vectors.
One-hot encoding represents each categorical value (here, a vocabulary word) as a binary vector of length V (vocabulary size) with a single 1 at the index of that word and 0s elsewhere. The vectors for different words are orthogonal.
From Tokens to Numbers
After tokenization, a sentence is a sequence of string tokens. Models cannot multiply strings by weights. The first design choice is a fixed vocabulary: an ordered list of unique tokens (often with a special <UNK> for out-of-vocabulary words). Each token maps to an integer index; one-hot encoding turns that index into a vector.
Linguistic units from Module 9.1.
Ordered word → index map.
Index → sparse binary vector.
Counts, TF-IDF, dense embeddings.
A Concrete Example
Vocabulary: ["cat", "dog", "mat", "sat"] with indices 0–3. Then:
| Word | Index | One-hot vector |
|---|---|---|
| cat | 0 | [1, 0, 0, 0] |
| dog | 1 | [0, 1, 0, 0] |
| mat | 2 | [0, 0, 1, 0] |
| sat | 3 | [0, 0, 0, 1] |
The sentence “cat sat” becomes two vectors: [1,0,0,0] then [0,0,0,1]. Document-level models often sum or OR these into a single bag vector—the seed of bag of words.
Code: sklearn and PyTorch
Properties That Matter
Dimensionality
- Length equals vocabulary size V.
- Large corpora → tens or hundreds of thousands of dims.
- Memory and compute grow with V.
Sparsity
- Exactly one nonzero entry per word vector.
- Most values are zero and carry no signal.
- Dot product of distinct words is always 0.
Semantics
- No notion of similarity.
- “king” and “queen” are as different as “king” and “banana”.
- Motivation for dense embeddings.
Strengths and Tradeoffs
Strengths
- Simple, deterministic, lossless for known vocabulary items.
- No training required; easy to debug.
- Natural input to linear models and early neural nets.
Tradeoffs
- Curse of dimensionality as V grows.
- Orthogonal vectors cannot capture synonymy or analogy.
- OOV words need special handling (
<UNK>).
“Integer token IDs are already numeric features.” Using raw indices (0, 1, 2, …) in a linear or distance-based model implies that index 5 is “close” to index 6—an artifact of vocabulary order, not meaning. One-hot (or an embedding layer) removes that false ordinal structure.
Where the Module Goes Next
One-hot vectors are the atomic building blocks. Aggregating them over a document yields bag-of-words counts; weighting those counts by rarity yields TF-IDF. Compressing distributional co-occurrence into dense vectors yields Word2Vec and friends. Keep one-hot in mind as the baseline every later method improves upon.
Knowledge Check
- Short Answer: What is the length of a one-hot word vector? Answer: The vocabulary size V.
- True/False: Distinct one-hot word vectors are orthogonal (dot product zero). Answer: True.
- Multiple Choice: One-hot encoding places a 1 at: (a) a random position, (b) the word’s vocabulary index, (c) the word frequency. Answer: (b).
- Short Answer: Why is Module 9.1 not enough for neural NLP by itself? Answer: Linguistic preprocessing yields tokens/symbols; models need numeric tensors.
- True/False: One-hot vectors encode semantic similarity between synonyms. Answer: False.
- Multiple Choice: Using raw integer IDs as features is problematic because: (a) they are too sparse, (b) they invent false ordinal relationships, (c) they require GPUs. Answer: (b).
- Short Answer: Name one special token often used for unknown words. Answer: <UNK> (or similar OOV marker).
- Short Answer: How many nonzero entries does a single word’s one-hot vector have? Answer: Exactly one.
- Multiple Choice: PyTorch’s
F.one_hotconverts: (a) strings to vectors, (b) integer indices to binary vectors, (c) embeddings to one-hots. Answer: (b). - True/False: Dense methods like Word2Vec were invented partly because one-hot is sparse and non-semantic. Answer: True.
Key Takeaways
- One-hot encoding maps each vocabulary word to a sparse binary vector of length V.
- It bridges Module 9.1’s linguistic preprocessing to numeric ML/NLP pipelines.
- Orthogonality means no built-in notion of word similarity.
- Sparsity and scale motivate TF-IDF weighting and dense embeddings next.
- Continue with TF-IDF to weight terms by importance.
Hands-on idea: Give a 10-word vocabulary and have students hand-write one-hot vectors for a three-word sentence, then verify with sklearn.
Discussion prompt: Estimate memory for a one-hot matrix of 50,000 words × 50,000 dims (float32) versus a 300-dimensional dense embedding table.
Recap: One-hot encoding turns discrete tokens into sparse numeric vectors—the starting point of Module 9.2. Next: TF-IDF.