Bayes Theorem — Module 2.3

How This Lesson Fits the Module

Conditional Probability taught you to read P(A | B) and chain evidence with the multiplication rule. But practitioners often know P(B | A)—the likelihood of data given a hypothesis—and need P(A | B)—the probability of the hypothesis given data. Bayes’ theorem is the bridge.

From spam filters and medical diagnosis to Bayesian neural networks and probabilistic programming, belief updating is central to AI. This lecture formalizes the theorem, introduces prior / likelihood / posterior vocabulary, and connects to Naive Bayes classification before the module turns to descriptive statistics (Mean, Variance, and beyond).

Learning Objectives

By the end of this lesson, students should be able to:

State Bayes’ theorem and identify prior, likelihood, evidence, and posterior.
Apply Bayes’ theorem to diagnostic and classification problems with concrete numbers.
Explain why base rates (priors) dramatically affect posterior conclusions.
Derive the Naive Bayes classifier from Bayes’ theorem plus conditional independence.
Distinguish frequentist point estimates from Bayesian belief updating.
Recognize the prosecutor’s fallacy and other misapplications of conditional probability.
Connect Bayes’ theorem to maximum a posteriori (MAP) estimation in ML.
Identify when approximate inference replaces exact posterior computation in modern AI.

Introduction: Inverting Conditional Probability

Doctors know how often a test is positive given disease—P(positive | disease). Patients need P(disease | positive). Spam filters estimate P(spam | email text) but generative models may store P(word | spam). Bayes’ theorem converts between these directions using the base rate P(hypothesis) and the marginal likelihood of the evidence.

In machine learning, Bayes’ theorem justifies:

Naive Bayes text classifiers
Bayesian optimization for hyperparameter search
Uncertainty quantification in Bayesian neural networks
MAP estimation as regularized maximum likelihood

Theorem — Bayes’ Theorem

For events A and B with P(B) > 0:

P(A | B) = P(B | A) · P(A) / P(B)

In hypothesis-testing language, with hypothesis H and evidence E:

Posterior = (Likelihood × Prior) / Evidence

P(H | E) = P(E | H) · P(H) / P(E)

Term	Symbol	Meaning	ML Analogy
Prior	P(H)	Belief before seeing evidence	Class frequency, weight prior
Likelihood	P(E \| H)	How probable is data if H is true	Generative model score
Evidence	P(E)	Overall probability of observing E	Normalizing constant
Posterior	P(H \| E)	Updated belief after evidence	Classifier output, updated weights

Worked Example: Rare Disease Testing

Suppose 1% of the population has a disease (prior P(D) = 0.01). A test has P(+ | D) = 0.99 (sensitivity) and P(+ | D^c) = 0.05 (false positive rate). A patient tests positive. What is P(D | +)?

First compute evidence: P(+) = P(+ | D)P(D) + P(+ | D^c)P(D^c) = 0.99(0.01) + 0.05(0.99) = 0.0594.

Then: P(D | +) = 0.99 × 0.01 / 0.0594 ≈ 0.167—only about 17% chance of disease despite a positive test. The low base rate dominates. This counterintuitive result is why AI systems must incorporate class imbalance and priors, not just accuracy on balanced benchmarks.

Frequentist View

Parameters are fixed unknowns
Data is random; repeat experiments
Point estimate via MLE
Confidence intervals over sampling

Bayesian View

Parameters have distributions (priors)
Data is observed; update beliefs
Posterior distribution over parameters
Credible intervals over parameters

Naive Bayes Classification

For class c and feature vector x = (x₁, …, x_n), Bayes’ theorem gives:

P(c | x) ∝ P(x | c) · P(c)

Naive Bayes assumes features are conditionally independent given c:

P(x | c) = ∏_i P(x_i | c)

The class with highest posterior wins. Training estimates P(x_i | c) from word counts (text) or histograms (continuous binned features). Despite naive assumptions, this remains a strong baseline for high-dimensional sparse data like email and document classification.

ML Example — Spam Filter Logic

P(spam | “winner”, “free”, “click”) ∝ P(“winner” | spam) · P(“free” | spam) · P(“click” | spam) · P(spam). Each likelihood is learned from labeled training emails. The prior P(spam) reflects base rate in the inbox or is tuned for precision/recall trade-offs.

Bridge from Conditional Probability Bayes’ theorem is a one-line rearrangement of P(A ∩ B) = P(B | A)P(A). The power is interpretive: it separates what we knew before (prior) from what the data says (likelihood).

Common Misconception: “A 99% accurate test means a positive result is 99% likely to be true.”

Reality: Accuracy conflates sensitivity and specificity. For rare conditions, most positives are false positives. Always compute the posterior with Bayes’ theorem using the true base rate P(D).

Common Misconception: “Bayesian methods are too slow for modern deep learning.”

Reality: Exact posteriors over millions of weights are intractable, but variational inference, MCMC, and Laplace approximations scale Bayesian ideas to large models. Bayesian optimization efficiently searches hyperparameters with far fewer trials than grid search.

MAP Estimation and Regularization

Maximum A Posteriori (MAP) estimation picks the parameter value that maximizes the posterior:

θ_MAP = argmax_θ P(θ | data) = argmax_θ P(data | θ) · P(θ)

With a Gaussian prior on weights, MAP corresponds to L2 regularization (weight decay) in neural networks. The prior pulls parameters toward zero; the likelihood pulls toward data fit. You have been doing approximate Bayesian inference every time you train with weight decay.

Knowledge Check

Short Answer: State Bayes’ theorem in words. Answer: Posterior equals likelihood times prior divided by evidence.
Computation: P(H) = 0.2, P(E | H) = 0.8, P(E) = 0.5. Find P(H | E). Answer: 0.8 × 0.2 / 0.5 = 0.32.
True/False: A low prior can make a positive test result weak evidence for a rare disease. Answer: True.
Multiple Choice: In Naive Bayes, P(x | c) is factored using: (a) unconditional independence, (b) conditional independence given c, (c) Bayes’ rule only, (d) no assumptions. Answer: (b).
Short Answer: What is the prosecutor’s fallacy? Answer: Confusing P(evidence | guilty) with P(guilty | evidence).
True/False: MAP estimation can be viewed as regularized maximum likelihood. Answer: True.
Computation: P(spam) = 0.4, P(word | spam) = 0.1, P(word | not spam) = 0.01. Which class has higher posterior for this single word (ignore normalization)? Answer: spam: 0.04 vs not spam: 0.006; spam wins.
Short Answer: What does P(E) normalize in Bayes’ theorem? Answer: It ensures posteriors over all hypotheses sum to 1.
Multiple Choice: Bayesian optimization is used primarily for: (a) image augmentation, (b) hyperparameter tuning, (c) batch normalization, (d) dropout. Answer: (b).
Short Answer: How does L2 weight decay relate to Bayesian inference? Answer: It corresponds to a Gaussian prior on weights in MAP estimation.

Key Takeaways

Bayes’ theorem inverts P(E | H) to P(H | E) using priors and the evidence normalizer.
Base rates matter: high test accuracy does not imply high posterior probability for rare events.
Naive Bayes applies Bayes’ theorem with a conditional-independence assumption over features.
MAP estimation connects Bayesian priors to regularization in neural network training.
Modern Bayesian ML uses approximate inference when exact posteriors are intractable.
Next: Mean begins the descriptive statistics arc—summarizing data before modeling full distributions.

Trainer’s Guide

Teaching strategy: Present the rare-disease example twice—once with 1% prevalence, once with 50%—to show how the same test yields radically different posteriors. Students remember the lesson when the arithmetic surprises them.

Hands-on idea: Implement a two-feature Naive Bayes spam filter on 20 labeled emails. Compare posterior P(spam | email) to sklearn’s MultinomialNB output.

Discussion prompt: When should a deployed classifier threshold be adjusted away from 0.5? How do priors and class imbalance interact?

What’s Next Continue to Mean to learn the most fundamental summary statistic for datasets and loss functions.