FastText

How This Lesson Fits the Module & Volume

Word2Vec and GloVe assign each word type a single atomic vector. That fails for typos, rare morphological variants, and true OOV tokens after Module 9.1 tokenization.

FastText (Bojanowski et al., 2017) extends Skip-gram-style training with character n-gram (subword) embeddings. A word vector is the sum of its subword vectors—so unseen words can still get a representation. It is the last major static word embedding before we lift to sentence embeddings and trainable layers.

Learning Objectives

By the end of this lesson, students should be able to:

Explain how FastText represents a word as a bag of character n-grams plus the word itself.
Describe why subwords help with morphology, typos, and OOV tokens.
Train FastText vectors with gensim and query an out-of-vocabulary word.
Compare FastText to Word2Vec and GloVe on OOV handling.
Identify when subword models still fall short (true contextual polysemy).
Connect FastText’s subword idea to later BPE/WordPiece tokenizers in transformers.

Definition

FastText learns embeddings for character n-grams and represents each word as the sum of the vectors of its n-grams (and usually a special vector for the full word). Training typically uses a Skip-gram with negative sampling objective on those enriched representations.

Subword Bags

For the word where with n-grams of length 3–6, FastText might include boundary-marked pieces like <wh, whe, her, ere>, …, plus <where>. Shared pieces let wherever and somewhere borrow statistical strength. After Module 9.1 stemming/lemmatization debates, FastText often reduces the need to aggressively normalize morphology for embedding quality.

Method	Unit of embedding	OOV behavior
Word2Vec / GloVe	Whole word type	UNK or drop
FastText	Word + char n-grams	Compose from subwords
Transformer tokenizers	Subword pieces (BPE etc.)	Always segmentable

Evolution Checkpoint

One-hot / BoW

Sparse identity & counts.

Word2Vec / GloVe

Dense type vectors.

FastText

Dense + subword composition.

Sentences / nn.Embedding

Longer units & task training.

Code: gensim FastText

from gensim.models import FastText

sentences = [
    ["fast", "text", "uses", "subword", "n", "grams"],
    ["morphology", "helps", "rare", "words"],
    ["typos", "like", "embeding", "still", "match"],
]
model = FastText(
    sentences,
    vector_size=50,
    window=3,
    min_count=1,
    min_n=3,          # min char n-gram length
    max_n=6,          # max char n-gram length
    sg=1,
    epochs=60,
)

print("embedding" in model.wv)           # False if never seen
print(model.wv["embedding"].shape)       # still works via n-grams!
print(model.wv.most_similar("subword", topn=3))

Compare: Static Embedding Family

Word2Vec

Atomic words.
Fast, simple.
Weak on OOV.

GloVe

Global counts.
Great pretrained sets.
Still atomic OOV.

Char n-grams.
OOV & morphology.
Larger model footprint.

Strengths and Tradeoffs

Strengths

Vectors for unseen morphological variants and typos.
Strong on morphologically rich languages.
Drop-in gensim API similar to Word2Vec.

Tradeoffs

More parameters (many n-grams).
Still not contextual: “bank” senses collide.
Character noise can sometimes hurt very frequent words.

Common Misconception

“FastText solves polysemy because it uses subwords.” Subwords help form (morphology/OOV), not sense. The finance and river senses of “bank” still share one composed vector. Contextual sentence encoders and transformers address sense.

Knowledge Check

Short Answer: What extra units does FastText embed beyond whole words? Answer: Character n-grams (subwords).
True/False: FastText can produce a vector for a word never seen in training. Answer: True (via n-grams).
Multiple Choice: FastText word vectors are typically: (a) one-hot, (b) sum of subword vectors, (c) TF-IDF rows. Answer: (b).
Short Answer: Name one problem FastText handles better than GloVe. Answer: OOV / typos / morphological variants.
True/False: FastText embeddings change with sentence context at inference. Answer: False—still static.
Multiple Choice: gensim class for this model is: (a) Word2Vec only, (b) FastText, (c) TfidfVectorizer. Answer: (b).
Short Answer: What do min_n and max_n control? Answer: Character n-gram length range.
Short Answer: How does FastText relate to later BPE tokenizers conceptually? Answer: Both exploit subword units for open vocabulary.
Multiple Choice: FastText does not by itself solve: (a) OOV typos, (b) contextual polysemy, (c) rare morphology. Answer: (b).
True/False: FastText training is often Skip-gram-like with negative sampling. Answer: True.

Key Takeaways

FastText composes word vectors from character n-grams.
It improves OOV and morphology over atomic Word2Vec/GloVe.
Vectors remain static and sense-agnostic.
gensim’s FastText API mirrors Word2Vec with min_n/max_n.
Next: lift from words to sentence embeddings.

Trainer’s Guide

Hands-on idea: Train tiny FastText, then compare model.wv["embedding"] vs. a deliberate typo "embeding" cosine similarity.

Discussion prompt: For a medical corpus full of rare drug names, when is FastText enough vs. when do you need domain pretraining or transformers?

Recap: FastText adds subword composition to static embeddings. Continue with Sentence Embeddings.