Probability introduced sample spaces, events, and the axioms that govern how uncertainty is measured. Real AI systems almost never ask “what is the probability of spam?” in isolation—they ask given this email’s features, what is the probability of spam? That shift from unconditional to conditional probability is the hinge of modern machine learning.
Conditional probability formalizes how new information narrows uncertainty. It underlies classification (P(class | features)), language modeling (P(token | context)), and every pipeline that updates beliefs as evidence arrives. This lecture builds the definition, the multiplication rule, and independence—the vocabulary needed for Bayes’ Theorem and the descriptive statistics lectures that follow.
Learning Objectives
By the end of this lesson, students should be able to:
- Define and compute conditional probability P(A | B) from joint and marginal probabilities.
- Apply the multiplication rule to chain conditional probabilities across multiple events.
- Distinguish independent events from merely correlated events.
- Interpret classifier outputs as conditional probabilities over labels given inputs.
- Use contingency tables and tree diagrams to organize conditional reasoning.
- Recognize when conditioning on the wrong variable induces bias (Simpson’s paradox preview).
- Connect conditional probability to Naive Bayes and generative classifiers.
- Identify common misconceptions about causation, correlation, and “updating” probability.
Introduction: Probability With Context
In Probability, you learned that P(A) measures how likely event A is before any specific evidence. In practice, models always condition on observed data. A fraud detector does not estimate P(fraud) in the abstract—it estimates P(fraud | transaction amount, location, velocity). A medical model estimates P(disease | symptoms and test results).
Conditional probability is the mathematical notation for “probability of A given that B has occurred.” It restricts the sample space to outcomes compatible with B, then re-normalizes so probabilities again sum to one. Every supervised learning classifier that outputs calibrated scores is, at its core, estimating a conditional distribution.
Let A and B be events with P(B) > 0. The conditional probability of A given B is:
P(A | B) = P(A ∩ B) / P(B)
Read as “the probability of A given B.” Intuitively: among all outcomes where B is true, what fraction also satisfy A?
Computing Conditional Probabilities
Start from a known joint distribution or counts in a dataset. The numerator counts outcomes in both A and B; the denominator counts all outcomes in B.
| Scenario | Given | Compute P(A | B) |
|---|---|---|
| Spam filter | P(spam ∩ contains “winner”) = 0.08, P(contains “winner”) = 0.10 | 0.08 / 0.10 = 0.80 |
| Medical test | 50 true positives out of 60 positive tests | 50 / 60 ≈ 0.83 (positive predictive value) |
| Die roll | A = even, B = roll > 3 | P(A|B) = P({4,6}) / P({4,5,6}) = 2/3 |
A neural network with softmax output layer produces P(y = k | x) for each class k. Training with cross-entropy loss directly optimizes these conditional probabilities against one-hot labels. The model learns P(label | features), not P(features | label)—though generative models flip that direction.
The Multiplication Rule
Rearranging the definition gives the multiplication rule:
P(A ∩ B) = P(B) · P(A | B) = P(A) · P(B | A)
For multiple events, chain the rule: P(A ∩ B ∩ C) = P(A) · P(B | A) · P(C | A ∩ B). Language models factor entire sentences this way—P(w1, w2, …, wn) as a product of next-token conditionals. That factorization is the backbone of autoregressive generation.
Unconditional P(A)
- Probability over the full sample space
- No observed evidence assumed
- Example: base rate of disease in population
- Prior belief before seeing data
Conditional P(A | B)
- Probability restricted to outcomes where B holds
- Evidence B is taken as given
- Example: disease probability after positive test
- Posterior belief after observing B
Independence
Events A and B are independent if knowing B does not change the probability of A:
P(A | B) = P(A) equivalently P(A ∩ B) = P(A) · P(B)
Naive Bayes classifiers assume features are conditionally independent given the class label—an approximation that trades realism for tractability. Despite the strong assumption, Naive Bayes often performs well on text classification because word presence patterns are weakly correlated once the topic is known.
Reality: They are generally different. P(disease | positive test) is not P(positive test | disease). Confusing the two is the prosecutor’s fallacy and leads to catastrophic errors in medical and legal reasoning. Bayes’ Theorem provides the correct way to invert conditional probabilities.
Reality: P(A | B) > P(A) means A and B are associated, not that B causes A. A hidden confounder C may explain both. In ML, spurious correlations in training data cause models to learn shortcuts that fail at deployment.
Conditional Probability in ML Pipelines
| Application | Conditional Quantity | Role |
|---|---|---|
| Binary classifier | P(positive | x) | Decision threshold, ROC curves |
| Naive Bayes | P(x | class) · P(class) | Generative classification via Bayes |
| Language model | P(token | context) | Next-token prediction, generation |
| Recommendation | P(click | user, item) | Ranking and personalization |
Knowledge Check
- Short Answer: What does P(A | B) mean in plain language? Answer: The probability of A among outcomes where B has occurred.
- Computation: P(A ∩ B) = 0.12, P(B) = 0.30. Find P(A | B). Answer: 0.12 / 0.30 = 0.40.
- True/False: If A and B are independent, P(A | B) = P(A). Answer: True.
- Multiple Choice: A softmax layer outputs: (a) unconditional probabilities, (b) conditional probabilities P(class | input), (c) priors only, (d) likelihoods only. Answer: (b).
- Short Answer: Write the multiplication rule for two events. Answer: P(A ∩ B) = P(B) · P(A | B).
- True/False: P(A | B) always equals P(B | A). Answer: False.
- Computation: Fair die: A = even, B = roll ≥ 4. Find P(A | B). Answer: P({4,6}) / P({4,5,6}) = 2/3.
- Short Answer: What independence assumption does Naive Bayes make about features? Answer: Features are conditionally independent given the class label.
- Multiple Choice: Conditioning requires: (a) P(A) = 0, (b) P(B) > 0, (c) A and B disjoint, (d) P(B) = 1. Answer: (b).
- Short Answer: How do language models factor sentence probability? Answer: As a product of conditional token probabilities given prior context.
Key Takeaways
- Conditional probability P(A | B) restricts the sample space to B and re-normalizes.
- The multiplication rule chains conditionals—essential for language models and sequential data.
- Independence means evidence does not change probability: P(A | B) = P(A).
- Classifiers estimate P(label | features); generative models often model P(features | label).
- P(A | B) and P(B | A) are different; Bayes’ theorem inverts between them.
- Correlation from conditioning does not imply causation—watch for confounders in training data.
- Next: Bayes’ Theorem formalizes belief updating with priors and likelihoods.
Teaching strategy: Use a 2×2 contingency table (disease / no disease × test positive / negative). Have students compute P(disease | positive) by hand before introducing Bayes’ theorem—they will feel the need for a systematic inversion rule.
Hands-on idea: Train a sklearn MultinomialNB on a tiny text corpus. Inspect predict_proba and relate each output to P(class | document features) under the naive independence assumption.
Discussion prompt: Why might P(fraud | transaction) differ sharply from P(transaction | fraud)? Which quantity does a deployed classifier need?