Distribution

How This Lesson Fits the Module

Probability introduced the language of chance. Mean, Variance, and Standard Deviation summarized samples—numbers computed from data you already observed. A distribution goes further: it describes how a random variable behaves across all possible outcomes, before or beyond any single dataset.

Every softmax output, every dropout mask, every noise term in diffusion models, and every prior in Bayesian inference ultimately references a distribution. Engineers who only know the mean and standard deviation of a batch miss the full picture: what shape does the randomness take?

If standard deviation measures spread, the distribution is the map of the entire landscape.

Learning Objectives

By the end of this lesson, students should be able to:

Define a random variable and distinguish discrete from continuous random variables.
Explain what a probability mass function (PMF) represents and state its key properties.
Explain what a probability density function (PDF) represents and why P(X = x) = 0 for continuous variables.
Define the cumulative distribution function (CDF) and relate it to PMFs and PDFs.
Compute simple probabilities from PMFs, PDFs, and CDFs in worked examples.
Connect distributions to descriptive statistics: mean as expectation, variance as spread of a distribution.
Recognize where distributions appear in ML: class probabilities, loss randomness, weight initialization, and generative models.
Identify common misconceptions about PDFs, normalization, and discrete–continuous mixing.

Introduction: Beyond the Sample

Suppose you train a classifier and record validation accuracy over 10 runs: 91.2%, 90.8%, 91.5%, … The mean and standard deviation of those 10 numbers summarize what you observed. But they do not tell you the law governing future runs. Is accuracy uniformly scattered? Clustered near 91%? Occasionally spiking to 95%?

A probability distribution answers that question. It assigns probabilities (or probability densities) to outcomes of a random variable—a numerical quantity whose value is determined by randomness. Distributions are the bridge between abstract probability theory and the stochastic behavior of real ML systems.

Random Variables

Definition — Random Variable

A random variable X is a function that maps outcomes of a random experiment to real numbers. We write X to denote the random quantity and x (lowercase) for a specific realized value.

Examples: X = number of misclassified images in a batch; X = pixel intensity after augmentation; X = final test loss after training.

Random variables fall into two fundamental types:

Type	Values	Example in ML	Described By
Discrete	Countable outcomes (finite or countably infinite)	Class label y ∈ {0, 1, …, 9}; token ID from a vocabulary	PMF
Continuous	Uncountable outcomes on intervals	Weight value w ∈ ℝ; augmented brightness in [0, 1]	PDF

Notation convention: P(X = x) for discrete probabilities; f(x) or p(x) for density/mass functions; F(x) for the CDF. Capital X denotes the random variable; lowercase x denotes a particular value.

Discrete Distributions: The PMF

Definition — Probability Mass Function (PMF)

For a discrete random variable X, the probability mass function is:

p(x) = P(X = x)

Properties: (1) p(x) ≥ 0 for all x; (2) ∑_x p(x) = 1 over all possible values of X.

The PMF gives the exact probability of each outcome. If X is the result of a fair six-sided die, p(1) = p(2) = … = p(6) = 1/6 and p(x) = 0 for all other x.

Example — Bernoulli PMF (Binary Classification)

A Bernoulli random variable models a single binary outcome: X ∈ {0, 1} with P(X = 1) = p and P(X = 0) = 1 − p.

p(x) = p^x(1 − p)^1−x, x ∈ {0, 1}

When a logistic classifier outputs “probability of class 1 = 0.73,” it is estimating the parameter p of a Bernoulli model for that instance. Cross-entropy loss is derived directly from this PMF.

Example — Categorical PMF (Multi-Class)

A categorical distribution generalizes Bernoulli to K classes. If X ∈ {1, 2, …, K} with probabilities p₁, p₂, …, p_K where ∑ p_k = 1:

p(x = k) = p_k

Softmax outputs form a categorical PMF over class labels. The vector [0.05, 0.80, 0.15] is a valid PMF: nonnegative entries summing to 1.

Continuous Distributions: The PDF

Definition — Probability Density Function (PDF)

For a continuous random variable X, the probability density function f(x) satisfies:

P(a ≤ X ≤ b) = ∫_a^b f(x) dx

Properties: (1) f(x) ≥ 0 for all x; (2) ∫_−∞^∞ f(x) dx = 1. For any single point, P(X = x) = 0.

This is the most common point of confusion for engineers: a PDF value is not a probability. The quantity f(x) is a density—only integrals over intervals yield probabilities. A density of 3.0 at some point is perfectly valid if the distribution is concentrated in a narrow region.

Example — Uniform on [0, 1]

If X is uniformly distributed on the interval [0, 1], then:

f(x) = 1 for 0 ≤ x ≤ 1, f(x) = 0 otherwise

P(0.2 ≤ X ≤ 0.5) = ∫_0.2^0.5 1 dx = 0.3. Weight initialization schemes often sample from uniform distributions over bounded intervals.

PMF (Discrete)

p(x) = P(X = x) is a probability.

Bar chart: bar heights are probabilities; they sum to 1.

PDF (Continuous)

f(x) is a density, not a probability.

Curve: area under the curve over an interval equals probability.

The CDF: One Function for All

Definition — Cumulative Distribution Function (CDF)

The CDF of a random variable X is:

F(x) = P(X ≤ x)

Properties: (1) F is non-decreasing; (2) lim_x→−∞ F(x) = 0; (3) lim_x→∞ F(x) = 1.

The CDF works for both discrete and continuous random variables—a unifying tool engineers should know well.

Variable Type	CDF from PMF/PDF	PMF/PDF from CDF
Discrete	F(x) = ∑_{t ≤ x} p(t)	p(x) = F(x) − F(x⁻), where x⁻ is the value just below x
Continuous	F(x) = ∫_−∞^x f(t) dt	f(x) = dF(x)/dx (where differentiable)

Worked Example — Uniform [0, 1] CDF

For X ~ Uniform(0, 1): F(x) = 0 for x < 0; F(x) = x for 0 ≤ x ≤ 1; F(x) = 1 for x > 1.

Then P(0.3 < X ≤ 0.7) = F(0.7) − F(0.3) = 0.7 − 0.3 = 0.4. For continuous variables, P(a < X ≤ b) = F(b) − F(a). Endpoints matter only at jump discontinuities (discrete case).

In ML evaluation, empirical CDFs appear constantly: “What fraction of examples have confidence below 0.9?” is a CDF question. Calibration plots compare predicted probabilities to empirical outcome frequencies—essentially auditing whether your model’s implied CDF matches reality.

Connecting Distributions to Mean and Standard Deviation

The summary statistics from earlier lectures are properties of distributions, not just samples:

Statistic	Distribution Form	Meaning
Mean (Expectation)	E[X] = ∑ x · p(x) or ∫ x f(x) dx	Center of mass of the distribution
Variance	Var(X) = E[(X − μ)²]	Average squared deviation from the mean
Standard Deviation	σ = √Var(X)	Spread in the same units as X (see Standard Deviation)

A sample mean x̄ and sample standard deviation s estimate the distribution’s μ and σ. Two datasets can share the same mean and standard deviation yet have completely different shapes—one symmetric, one skewed. That is why we need the full distribution, not just first- and second-moment summaries.

Engineering Principle

When debugging a model, ask: “What distribution am I assuming, and what distribution am I actually observing?” A mismatch between assumed and empirical distributions—in labels, features, gradients, or activations—is a frequent root cause of poor convergence and miscalibrated predictions.

Distributions in Machine Learning

Input data — Features and labels follow (often unknown) distributions ↓ Model outputs — Softmax produces a categorical PMF over classes ↓ Loss computation — Negative log-likelihood assumes a distributional form ↓ Stochastic training — Mini-batch sampling, dropout masks, data augmentation noise ↓ Initialization — Weights drawn from chosen distributions (uniform, normal) ↓ Generative models — Explicitly learn or sample from target distributions

Three high-frequency distributions in production ML:

Bernoulli / Categorical — Classification heads and language-model token prediction.
Gaussian (Normal) — Weight initialization, noise models, latent spaces in VAEs. Covered in depth in the next lecture.
Uniform — Bounded random initialization and exploration in reinforcement learning.

Coming UpThe Gaussian / Normal Distribution lecture develops the most important continuous distribution in AI—the bell curve that governs initialization, aggregation, and normalization.

Visual Intuition: PMF, PDF, and CDF Together

Consider a discrete variable X with outcomes {0, 1, 2} and PMF p(0) = 0.2, p(1) = 0.5, p(2) = 0.3:

PMF bars at heights 0.2, 0.5, 0.3.
CDF steps: F(0) = 0.2, F(1) = 0.7, F(2) = 1.0.

For a continuous variable, imagine a smooth bell-shaped PDF. The CDF is the accumulated area from the left: an S-shaped curve climbing from 0 to 1. The PDF is the slope of the CDF at each point.

Common Misconceptions

Misconception 1: “f(x) is the probability that X equals x.”

Why people believe it: The discrete PMF p(x) = P(X = x) works exactly this way, so the parallel notation f(x) feels identical.

Reality: For continuous X, P(X = x) = 0 always. Only interval probabilities from integration are valid: P(a ≤ X ≤ b) = ∫_a^b f(x) dx.

Misconception 2: “A PDF can have values greater than 1.” is impossible.

Why people believe it: Probabilities are bounded by 1, so densities should be too.

Reality: Densities are not probabilities. A Uniform(0, 0.1) distribution has f(x) = 10 on [0, 0.1]—the total area still integrates to 1 because the interval is narrow.

Misconception 3: “Discrete and continuous are just implementation details—I can always treat data as continuous.”

Why people believe it: Neural networks use floating-point tensors for everything, including class indices stored as floats.

Reality: Class labels are discrete objects requiring PMFs (softmax + cross-entropy). Treating them as continuous and applying regression loss is a modeling error, not a harmless shortcut.

Misconception 4: “The CDF and PDF are unrelated.”

Why people believe it: Textbooks present them in separate sections with different formulas.

Reality: For continuous variables, F(x) is the integral of f(t) and f(x) is the derivative of F(x). They are two views of the same distribution.

Quick Knowledge Check

Short Answer: What is a random variable? Answer: A function mapping random experiment outcomes to real numbers.
True/False: For a continuous random variable, P(X = 5.0) can be positive. Answer: False — point probabilities are zero; only intervals have positive probability.
Multiple Choice: Which function applies to discrete random variables? (a) PDF, (b) PMF, (c) CDF only, (d) Gradient. Answer: (b) PMF.
Short Answer: State two properties of a valid PMF. Answer: p(x) ≥ 0 for all x; ∑ p(x) = 1.
Computation: A fair coin: X = 1 (heads), X = 0 (tails). What is p(1)? Answer: 0.5.
Short Answer: Write the CDF F(x) in terms of the PDF f(t). Answer: F(x) = ∫_−∞^x f(t) dt.
True/False: A softmax output vector is a valid PMF over classes. Answer: True — nonnegative entries summing to 1.
Multiple Choice: For Uniform(0, 1), what is P(0 ≤ X ≤ 0.25)? (a) 0.25, (b) 1.0, (c) 0, (d) 0.5. Answer: (a) 0.25.
Short Answer: How does the CDF relate to computing P(a < X ≤ b) for continuous X? Answer: P(a < X ≤ b) = F(b) − F(a).
True/False: Mean and standard deviation fully determine the shape of any distribution. Answer: False — different shapes can share the same mean and SD.

Key Takeaways

A distribution describes all possible outcomes of a random variable—not just a single sample.
Discrete variables use PMFs (probabilities at points); continuous variables use PDFs (densities requiring integration).
The CDF F(x) = P(X ≤ x) unifies both types and computes interval probabilities.
Mean, variance, and standard deviation are properties derived from the underlying distribution.
Softmax outputs are categorical PMFs; cross-entropy loss encodes Bernoulli/categorical assumptions.
PDF values are not probabilities—only areas under the curve are.
Debugging ML systems often means comparing assumed distributions to empirical ones.
The Gaussian distribution—covered next—is the dominant continuous distribution in deep learning practice.