Module 2.2: Calculus gave you the dynamic story of learning—Derivatives and Gradient Descent adjust parameters to minimize loss. The capstone, Optimization, framed training as navigating a loss landscape toward a best setting. Machine learning, however, does not only fit parameters; it must reason under uncertainty.
Probability is the first tool in Module 2.3: Probability & Statistics for that uncertain world. It assigns numbers in [0, 1] to outcomes and events, governed by axioms every valid model must respect. This lecture builds the discrete foundation—sample spaces, events, P(A)—that later lectures extend to conditioning, Bayes’ rule, and distributions.
If calculus tells you how to move across a loss surface, probability tells you how to interpret predictions when the data-generating process is noisy.
Learning Objectives
By the end of this lesson, students should be able to:
- Define a sample space and an event, and express compound events with set operations.
- State the three Kolmogorov axioms and explain why they constrain any valid probability model.
- Compute P(A) for discrete sample spaces using equally likely outcomes or a probability mass function.
- Distinguish theoretical probability from empirical (frequentist) estimates from data.
- Interpret softmax outputs and classification scores as approximate class probabilities.
- Apply the complement rule P(Ac) = 1 − P(A) and finite additivity for mutually exclusive events.
- Connect probability to ML evaluation: accuracy, error rate, and calibrated confidence.
- Recognize when treating model outputs as probabilities requires calibration or is merely heuristic.
Introduction: From Point Estimates to Uncertainty
At the end of Module 2.2, Optimization framed learning as minimizing a loss over parameters. The output of training is concrete: weights, biases, a decision boundary. But when a spam filter labels an email, when a medical model flags an X-ray, or when a language model samples the next token, the engineer must answer a deeper question: how sure are we?
Probability formalizes “how sure” as a number in [0, 1]. Zero means impossible (in the model’s world); one means certain. Everything in between quantifies partial belief or long-run frequency. Machine learning uses both views—frequentist evaluation on held-out data and Bayesian updating in later lectures—but they all rest on the same axiomatic core introduced here.
Sample Spaces and Events
An experiment or random process has outcomes we cannot fully predict in advance. The set of all possible outcomes is the sample space, denoted Ω (omega).
The sample space Ω is the set of all elementary outcomes of a random experiment.
An event A is a subset of Ω—a collection of outcomes we treat as a single statement (“the email is spam,” “the die shows an even number”).
- Binary spam filter: Ω = {spam, not spam}. Event S = {spam}.
- 3-class image model: Ω = {cat, dog, bird}. Event C = {cat}.
- Regression (binned): Ω = price brackets; event “price > $500k” is a union of upper brackets.
Events combine with standard set operations: union (A ∪ B, at least one occurs), intersection (A ∩ B, both occur), and complement (Ac, A does not occur). In code, these mirror logical OR, AND, and NOT on boolean masks over outcomes or labels.
The Probability of an Event: P(A)
P(A) assigns a single number to event A measuring how likely A is. For a finite sample space with equally likely outcomes, the classical definition applies:
P(A) = |A| / |Ω|
where |A| counts outcomes in A. For unequal likelihoods—the normal case in ML—each outcome ω carries a weight p(ω) with ∑ω∈Ω p(ω) = 1, and P(A) = ∑ω∈A p(ω).
| Setting | How P(A) Is Determined | ML Example |
|---|---|---|
| Equally likely outcomes | Count favorable outcomes / total outcomes | Balanced dataset: P(random example is class k) = 1/K |
| Given PMF | Sum p(x) over outcomes in A | Softmax vector p over K classes |
| Empirical frequency | Count occurrences in data / n | Training set class prior: 12% positive labels |
The Kolmogorov Axioms
Any assignment P(·) used in theory or software must satisfy three axioms. Violating them—even silently in a pipeline—produces incoherent confidence scores.
- Non-negativity: P(A) ≥ 0 for every event A.
- Normalization: P(Ω) = 1.
- Countable additivity: For pairwise disjoint events A1, A2, …, P(∪i Ai) = ∑i P(Ai).
For finite disjoint A and B: P(A ∪ B) = P(A) + P(B).
Immediate consequences used daily in engineering:
- Complement rule: P(Ac) = 1 − P(A).
- General union: P(A ∪ B) = P(A) + P(B) − P(A ∩ B).
- Bounds: 0 ≤ P(A) ≤ 1 always.
Loss Minimization (Module 2.2)
- Output: optimal parameters w*
- Question: “What setting minimizes error?”
- Single best point on a surface
- Gradient descent drives updates
Probability (Module 2.3)
- Output: distribution over outcomes
- Question: “How likely is each outcome?”
- Mass spread across the sample space
- Axioms constrain valid assignments
Discrete Probability
Most introductory ML problems begin with discrete outcomes: class labels, token IDs, bucketed counts. A discrete random variable X takes values in a finite or countably infinite set with a probability mass function (PMF):
p(x) = P(X = x), with ∑x p(x) = 1
Fair six-sided die: Ω = {1, 2, 3, 4, 5, 6}, p(k) = 1/6. Event “even” = {2, 4, 6}, so P(even) = 3/6 = 1/2.
Imbalanced fraud detection: Ω = {fraud, legitimate}, p(fraud) = 0.02 from historical data. Then P(legitimate) = 1 − 0.02 = 0.98 by the complement rule.
Classification Confidence in Machine Learning
A classifier maps features x to a label. Hard prediction picks argmaxk p(k | x). Soft prediction returns the full vector p = (p1, …, pK), ideally satisfying the axioms on {1, …, K} for each fixed x:
- Each pk ≥ 0
- ∑k pk = 1
The softmax function converts logits z into such a vector:
softmax(zk) = ezk / ∑j ezj
Cross-entropy loss from Module 2.2 encourages predicted p(y | x) to match the one-hot true label. High softmax mass on the correct class means high stated confidence—but that is not the same as calibrated probability (reliability diagrams and temperature scaling address that gap in production).
| Quantity | Formula / Interpretation | Engineering Use |
|---|---|---|
| Class prior P(Y = k) | Fraction of label k in population or training set | Baseline accuracy; cost-sensitive thresholds |
| Predicted P(Y = k | x) | Model output after softmax or calibrated head | Ranking, abstention, human-in-the-loop routing |
| Error rate | P(predicted ≠ true) on test data | Empirical complement of accuracy |
Always check that predicted probabilities sum to one and are non-negative before using them in downstream decisions (expected cost, Bayesian model combination, or A/B test analysis). A bug that outputs negative logits through the wrong activation can silently violate the axioms.
Reality: Softmax outputs are relative scores normalized to sum to 1. Modern classifiers are often overconfident—stated 99% may correspond to a lower true hit rate on held-out data. Treat raw scores as rankings unless you verify calibration on a representative validation set.
Reality: Empirical frequency estimates priors from data, but P(A | x) is a conditional quantity that depends on features. The prior P(spam) = 0.4 does not equal P(spam | this email’s content).
Knowledge Check
- Short Answer: What is a sample space? Answer: The set Ω of all possible elementary outcomes of a random experiment.
- True/False: For any event A, P(A) can be negative if the model is uncertain. Answer: False — axiom of non-negativity requires P(A) ≥ 0.
- Computation: A fair coin is tossed twice. What is P(at least one head)? Answer: Outcomes {HH, HT, TH, TT}; favorable three; P = 3/4.
- Multiple Choice: If P(A) = 0.35, then P(Ac) is: (a) 0.35, (b) 0.65, (c) 1.35, (d) 0. Answer: (b).
- Short Answer: State the normalization axiom. Answer: P(Ω) = 1.
- Computation: PMF: P(X = 0) = 0.1, P(X = 1) = 0.4, P(X = 2) = 0.5. Find P(X ≤ 1). Answer: 0.1 + 0.4 = 0.5.
- Short Answer: How does softmax ensure axioms for class probabilities? Answer: Exponentials are positive; dividing by their sum yields nonnegative values that add to 1.
- True/False: Disjoint events A and B can have P(A ∪ B) = P(A) + P(B). Answer: True.
- Multiple Choice: In a 3-class problem, a valid probability vector is: (a) (0.5, 0.5, 0.5), (b) (−0.1, 0.6, 0.5), (c) (0.2, 0.3, 0.5), (d) (1, 1, 1). Answer: (c).
- Short Answer: Why does Module 2.3 follow calculus in the curriculum? Answer: Optimization gives point estimates; probability quantifies uncertainty and supports inference, evaluation, and generative modeling.
Key Takeaways
- The sample space Ω lists all outcomes; events are subsets of Ω.
- P(A) quantifies how likely event A is, always between 0 and 1.
- Kolmogorov’s axioms: non-negativity, P(Ω) = 1, and additivity over disjoint events.
- Discrete PMFs assign masses p(x) that sum to 1.
- Softmax produces valid discrete distributions over classes but may need calibration for decision-making.
- Empirical frequencies from data estimate probabilities; models target conditional distributions P(label | features).
- Module 2.2 optimization and Module 2.3 probability answer complementary questions about ML systems.
- Next: Conditional Probability refines P(A) to P(A | B)—probability when partial information is known.
Teaching strategy: Start with a concrete classifier outputting three softmax values on the board. Verify they sum to 1, then ask what question each number answers. Only then introduce Ω and axioms—notation follows intuition.
Hands-on idea: Have students compute empirical class priors from a small CSV, then compare to softmax outputs on the same examples. Discuss mismatch between prior and conditional predictions.
Discussion prompt: After Optimization, we trust loss minima. When should we trust probability outputs instead of argmax labels?
Expected difficulty: Students conflate P(A) with P(A | B). Preview that conditioning is the subject of the next lecture.