Naive Bayes — Module 5.2

How This Lesson Fits the Module

Tree ensembles like random forest learn complex boundaries from data. Naive Bayes takes the opposite path: a simple probabilistic model with a strong independence assumption. It trains in one pass, handles high-dimensional text, and often beats heavier models as a spam-filter or sentiment baseline.

The “naive” assumption—features are independent given the class—is usually wrong, yet the classifier often works remarkably well.

Learning Objectives

By the end of this lesson, students should be able to:

State Bayes’ rule and the naive conditional independence assumption.
Choose among GaussianNB, MultinomialNB, and BernoulliNB.
Build a text-classification pipeline with CountVectorizer or TfidfVectorizer.
Apply Laplace smoothing via alpha to handle unseen tokens.
Compare training speed and memory use to tree and SVM models.

Bayes’ Rule in One Line

Pick the class c that maximizes P(c | x) ∝ P(c) · ∏ P(xᵢ | c). The prior P(c) is class frequency; each likelihood P(xᵢ | c) is estimated from training counts or distributions. Naive Bayes ignores feature correlations—hence the name.

Variant	Feature type	Typical use
`GaussianNB`	Continuous (normal per class)	Simple sensor or biometric data
`MultinomialNB`	Non-negative counts (word frequencies)	Text classification, bag-of-words
`BernoulliNB`	Binary presence/absence	Short text, keyword flags
`ComplementNB`	Imbalanced text	Skews correction for rare classes

Text Classification Pipeline

The 20 newsgroups subset is ideal for demonstrating Multinomial Naive Bayes with TF–IDF weighting—fast to train and surprisingly competitive.

from sklearn.datasets import fetch_20newsgroups
from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
from sklearn.metrics import classification_report

categories = ["sci.space", "rec.sport.baseball", "talk.politics.misc"]
data = fetch_20newsgroups(subset="train", categories=categories, shuffle=True, random_state=42)

X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, stratify=data.target, random_state=42
)

text_clf = Pipeline([
    ("tfidf", TfidfVectorizer(max_features=10_000, stop_words="english")),
    ("clf", MultinomialNB(alpha=0.1)),
])
text_clf.fit(X_train, y_train)
print(classification_report(y_test, text_clf.predict(X_test),
      target_names=data.target_names))

Smoothing and Priors

alpha adds pseudo-counts so zero-frequency words do not zero out entire class probabilities. Tune it on validation data. Use fit_prior=False when class frequencies in production differ sharply from training.

Engineering Habit — Baseline Before BERT

Ship a TF–IDF + MultinomialNB baseline in hours. Measure latency and F1 before investing in transformer fine-tuning. Many inbox spam filters still rely on variants of this stack.

Strengths

Extremely fast training and prediction
Works well with thousands of sparse features
Natural probabilistic outputs
Small memory footprint

Weaknesses

Independence assumption ignores word order
Poor with correlated numeric features
Calibrated probabilities may need Platt scaling
Not competitive on complex vision/audio tasks

Critical Mistake — GaussianNB on Raw Counts

Applying GaussianNB to sparse word counts violates its normality assumption. Match the variant to the data: MultinomialNB for counts, BernoulliNB for binary bags, GaussianNB for continuous sensors.

Knowledge Check

Short Answer: What makes Naive Bayes “naive”? Answer: It assumes features are conditionally independent given the class.
True/False: MultinomialNB requires TF–IDF; raw counts are invalid. Answer: False—counts work; TF–IDF often helps but is not required.
Multiple Choice: Laplace smoothing parameter: (a) alpha, (b) C, (c) max_depth. Answer: (a).
Short Answer: Why is Naive Bayes popular for spam? Answer: High-dimensional sparse text, fast updates, decent accuracy with simple math.
Short Answer: What does stop_words="english" remove? Answer: Common words like “the” and “is” that carry little class signal.
True/False: GaussianNB assumes features follow a Gaussian per class. Answer: True.
Multiple Choice: BernoulliNB is a natural fit for: (a) continuous heights, (b) binary presence/absence features, (c) images only. Answer: (b).
Short Answer: What does Laplace (alpha) smoothing prevent? Answer: Zero probabilities for unseen feature–class combinations.
True/False: Feature dependence always makes Naive Bayes unusable. Answer: False—it can still be a strong baseline despite the assumption.
Multiple Choice: TfidfVectorizer + MultinomialNB is classic for: (a) time-series forecasting, (b) text classification, (c) k-means. Answer: (b).

Key Takeaways

Naive Bayes applies Bayes’ rule with a conditional independence assumption.
Pick Gaussian, Multinomial, or Bernoulli to match feature types.
Text pipelines pair vectorizers with MultinomialNB for fast baselines.
Use alpha smoothing for unseen tokens and sparse data.
Next: KNN for instance-based, distance-driven classification.

Trainer’s Guide

Hands-on idea: Students train MultinomialNB and a linear SVM on the same newsgroups split, compare train time, inference latency, and macro-F1.

Discussion prompt: Which words does tfidf.get_feature_names_out() with highest NB log-probability ratios reveal about each category?

Recap: Naive Bayes applies Bayes’ rule with a conditional-independence assumption—fast and strong on text. Continue with KNN.