Partial Derivatives

How This Lesson Fits the Module

In Derivatives, you learned how a single-variable function changes at one point: the derivative f′(x) measures instantaneous rate of change along one axis. Real ML systems almost never depend on a single number. A neural network’s loss depends on millions of weights, input features, and hyperparameters simultaneously.

Partial derivatives extend the derivative idea to multivariable functions. They answer: “If I nudge one input while holding everything else fixed, how does the output change?” That question is the atomic unit of backpropagation—every weight update in training is driven by a partial derivative of the loss.

This lecture bridges single-variable calculus to the machinery you will need in Chain Rule, Gradient, and Gradient Descent. If derivatives tell you how fast a curve rises, partial derivatives tell you how fast a loss surface rises in each coordinate direction.

Learning Objectives

By the end of this lesson, students should be able to:

Explain why ML loss functions are multivariable and why single-variable derivatives are insufficient for training.
Define the partial derivative ∂f/∂x as a limit with all other variables held constant.
Compute partial derivatives of polynomial and elementary multivariable functions by treating other variables as constants.
Interpret ∂f/∂x geometrically as the slope of a slice through a surface.
Describe what “holding other variables fixed” means both algebraically and in code.
Introduce the Jacobian matrix as the organized collection of all first-order partial derivatives of a vector-valued function.
Connect partial derivatives to per-weight sensitivity in a neural network loss.
Recognize common notation variants (∂, ∇, subscripts) used in ML papers and frameworks.

Introduction: One Input Is Not Enough

Consider a linear model with two weights predicting house price from square footage and bedroom count:

ŷ = w₁x₁ + w₂x₂ + b

Training minimizes a loss L that depends on w₁, w₂, and b at once—not on a single variable. A deep network is the same idea at scale: the loss L(θ) is a function of a parameter vector θ ∈ ℝ^d where d may be 10⁶ or more.

A multivariable function maps several inputs to one or more outputs:

Scalar output: f(x, y) = x² + 3xy + y² — a loss or energy function
Vector output: f(x) = [f₁(x), …, f_m(x)]^T — a layer’s activations before a nonlinearity

Partial derivatives are how calculus handles the first case directly and builds the second through the Jacobian.

From Derivatives to Partial Derivatives

Recall from Derivatives that for f: ℝ → ℝ,

f′(x) = lim_Δx→0 [f(x + Δx) − f(x)] / Δx

For f: ℝⁿ → ℝ, we cannot ask for a single derivative—there are n independent directions. Instead we define n partial derivatives, one per input coordinate.

Definition — Partial Derivative with Respect to x

Let f(x, y, …, z) be a real-valued function. The partial derivative of f with respect to x is:

∂f/∂x = lim_Δx→0 [f(x + Δx, y, …, z) − f(x, y, …, z)] / Δx

All variables except x are held fixed during the limit. Notation variants: f_x, D_xf, or ∂_xf.

Similarly, ∂f/∂y differentiates with respect to y while holding x, z, … constant. Each partial derivative measures sensitivity along one coordinate axis of the input space.

The Core Rule: Hold Everything Else Fixed

The phrase “holding other variables fixed” is not a metaphor—it is a precise computational instruction.

Algebraic View

To compute ∂f/∂x for f(x, y) = x²y + 5y:

Treat y as a constant: f looks like y·x² + 5y
Differentiate with respect to x only: ∂f/∂x = 2xy
y is still a variable in the result—only frozen during differentiation

Geometric View

Fix y = y₀. The slice z = f(x, y₀) is an ordinary single-variable curve in the xz-plane.

∂f/∂x evaluated at (x₀, y₀) is the slope of that slice—how steep the surface is if you walk purely in the x direction.

∂f/∂y is the slope of a slice with x fixed.

Engineering Principle

When PyTorch computes loss.backward(), it evaluates partial derivatives of the scalar loss with respect to each tensor element—each weight—while treating the computational graph’s other paths as fixed. Automatic differentiation implements the “hold fixed” rule at scale.

Worked Examples

Example 1 — Polynomial in Two Variables

Let f(x, y) = x³ + 2xy² − 7y.

∂f/∂x: Treat y as constant → 3x² + 2y²

∂f/∂y: Treat x as constant → 4xy − 7

At (x, y) = (1, 2): ∂f/∂x = 3 + 8 = 11 and ∂f/∂y = 8 − 7 = 1. Near this point, f is far more sensitive to x than to y.

Example 2 — Mean Squared Error (Two Weights)

For one training example with target t and prediction ŷ = w₁x₁ + w_{2x₂, squared error is:}

L(w₁, w₂) = (ŷ − t)²

Let e = ŷ − t. By the chain rule preview (full treatment in Chain Rule):

∂L/∂w₁ = 2e · x₁, ∂L/∂w₂ = 2e · x₂

Each partial derivative asks: “How does loss change if I adjust only this weight?” The answer depends on the feature connected to that weight—exactly what gradient-based learning exploits.

Notation in ML Literature

Symbol	Meaning	Typical Context
∂f/∂x_i	Partial derivative w.r.t. the ith input	Calculus texts, loss derivatives per feature
∇_xf	Gradient — vector of all partial derivatives	Optimization, Gradient lecture
∂L/∂θ_j	Partial derivative of loss w.r.t. weight j	Backpropagation, parameter updates
J_f or ∂f/∂x	Jacobian matrix of vector function f	Layer Jacobians, change-of-variables
`tensor.grad`	Stored partial derivative(s) in PyTorch	After `loss.backward()`

Coming UpThe gradient ∇f packages all partial derivatives of a scalar function into one vector—covered systematically in Gradient. The chain rule explains how to compute them through composed layers in Chain Rule.

Loss with Many Weights

A neural network with d trainable parameters defines a loss function:

L: ℝ^d → ℝ, θ = (θ₁, θ₂, …, θ_d)

Training requires d partial derivatives—one per weight:

∂L/∂θ₁, ∂L/∂θ₂, …, ∂L/∂θ_d

Each answers a local sensitivity question: holding all other weights fixed, should I increase or decrease θ_j to reduce loss, and how strongly?

Scale	Parameters d	Partial Derivatives Needed	Computed By
Toy linear model	3–10	Hand calculation feasible	Pen and paper
Small MLP on MNIST	≈ 10⁴–10⁵	One per weight	Autograd (PyTorch, JAX)
Large language model	≈ 10⁹–10¹²	One per parameter	Distributed autograd + optimizers

Humans never write billions of partial derivatives by hand. Engineers instead build differentiable programs—computational graphs where every operation has known local derivatives. Partial derivatives are then accumulated via the chain rule. The conceptual shift from high school calculus to ML engineering is: you define the function; the framework computes every ∂f/∂x_i.

Example 3 — Per-Weight Gradients in PyTorch

import torch

w1 = torch.tensor(0.5, requires_grad=True)
w2 = torch.tensor(-1.0, requires_grad=True)
x1, x2, target = 2.0, 3.0, 1.0

y_hat = w1 * x1 + w2 * x2
loss = (y_hat - target) ** 2

loss.backward()  # computes ∂L/∂w1 and ∂L/∂w2

print(w1.grad)   # ∂L/∂w1 at current (w1, w2)
print(w2.grad)   # ∂L/∂w2

Each .grad attribute stores exactly one partial derivative—the sensitivity of loss to that scalar parameter, with all other parameters treated as fixed during the local computation.

Introduction to the Jacobian

When a function outputs a vector, partial derivatives organize into a matrix called the Jacobian.

Definition — Jacobian Matrix

For f: ℝⁿ → ℝ^m with components f₁, …, f_m, the Jacobian J_f(x) is the m × n matrix:

J_f = ∂f/∂x, [J]_i,j = ∂f_i/∂x_j

Row i contains partial derivatives of output i; column j contains sensitivities of all outputs to input j.

Special case: If f: ℝⁿ → ℝ is scalar (like a loss), the Jacobian is a row vector of length n—the transpose of the gradient ∇f.

Example 4 — Jacobian of a 2D Vector Function

Let f(x, y) = [x² + y, xy]^T. Then:

J_f = ∂f₁/∂x ∂f₁/∂y ∂f₂/∂x ∂f₂/∂y = 2x 1 y x

In a neural network, each layer is a vector function of its inputs. The Jacobian of a layer describes how small input perturbations propagate to outputs—the linearization used in advanced stability analysis and normalizing flows.

Why the Jacobian Matters in AI

Backpropagation multiplies Jacobians (or Jacobian-vector products) along the computational graph—efficiently, without forming full matrices for huge layers.
Generative models (normalizing flows) require Jacobians for change-of-variables: log-det J adjusts probability densities.
Dynamical systems (RNNs, ODE nets): eigenvalues of a Jacobian near a fixed point predict whether perturbations explode or decay—connecting to eigenanalysis in Eigenvalues.

Introductory ML courses emphasize scalar loss and per-weight partial derivatives; research and advanced architecture work routinely invoke full Jacobians. Know both levels.

Visualizing a Loss Surface

With two weights, L(w₁, w₂) can be visualized as a bowl-shaped surface over the w₁w₂-plane. At a point (w₁*, w₂*):

∂L/∂w₁ > 0 → increasing w₁ (alone) increases loss → move w₁ downward
∂L/∂w₁ < 0 → increasing w₁ decreases loss → move w₁ upward
∂L/∂w₁ = 0 → locally flat in the w₁ direction (may still slope in w₂)

Fix all inputs except one — algebraic “hold constant” rule ↓ Compute ordinary derivative of the slice — ∂f/∂x_i ↓ Collect all coordinates — gradient vector ∇f (next lecture) ↓ Stack for vector outputs — Jacobian matrix J ↓ Scale to millions of weights — autograd + chain rule

In high dimensions the surface cannot be drawn, but the logic is identical: each partial derivative is one coordinate of the steepest-local-information vector field that drives learning.

Higher-Order Partials (Brief)

Partial derivatives can themselves be differentiated. The second partial ∂²f/∂x_i∂x_j measures curvature—how the slope in direction i changes when moving in direction j. For smooth functions, Clairaut’s theorem guarantees mixed partials commute: ∂²f/∂x∂y = ∂²f/∂y∂x.

The matrix of second partials of a scalar function is the Hessian — central to Newton’s method and sharpness analysis of minima. Full treatment appears in Optimization. For now, recognize that first-order partials (this lecture) supply the per-coordinate slopes; second-order partials supply curvature.

Common Misconceptions

Misconception 1: “∂f/∂x is just the regular derivative with extra notation.”

Why people believe it: The computation rules look identical.

Reality: The definition explicitly freezes other variables. f(x, y) = xy has ∂f/∂x = y, not something involving both variables’ rates in a single scalar derivative. Multivariable structure matters.

Misconception 2: “If ∂f/∂x = 0, the function is flat.”

Why people believe it: In one variable, zero derivative often means a critical point.

Reality: Zero partial in x means flat only along the x direction. The surface may still slope in y, z, … A saddle point can have ∂f/∂x = 0 and ∂f/∂y ≠ 0.

Misconception 3: “We need the full Jacobian matrix at every training step.”

Why people believe it: Jacobians appear in theory-heavy treatments of backprop.

Reality: Standard reverse-mode autograd computes Jacobian-vector products (∂L/∂θ) without materializing the full Jacobian for billion-parameter models. The Jacobian is the organizing concept; efficient training uses structure, not dense matrices.

Misconception 4: “Partial derivatives tell you the direction of steepest ascent.”

Why people believe it: They are slopes, and slopes feel directional.

Reality: A single partial is only the slope along one axis. The steepest direction requires combining all partials into the gradient vector ∇f—the subject of the next module lectures.

Quick Knowledge Check

Short Answer: What does “holding other variables fixed” mean when computing ∂f/∂x? Answer: Treat all variables except x as constants during differentiation.
Computation: For f(x, y) = x²y + 3x, find ∂f/∂x and ∂f/∂y. Answer: ∂f/∂x = 2xy + 3; ∂f/∂y = x².
True/False: A neural network loss with 1 million weights requires 1 million partial derivatives for gradient-based training. Answer: True (one ∂L/∂θ_j per parameter).
Multiple Choice: The Jacobian of f: ℝ³ → ℝ² has shape: (a) 3×2, (b) 2×3, (c) 3×3, (d) 2×2. Answer: (b) 2×3 — m outputs × n inputs.
Short Answer: What does ∂L/∂w_j tell an optimizer? Answer: How much and in which direction (sign) loss changes when weight w_j is perturbed, holding other weights fixed.
True/False: For scalar f, the gradient ∇f is the transpose of the Jacobian row vector. Answer: True.
Computation: For f(x, y) = sin(xy), find ∂f/∂x. Answer: y cos(xy) (chain rule on the outer sin, inner xy with y fixed).
Short Answer: Why can engineers train billion-parameter models without hand-computing partials? Answer: Automatic differentiation computes them via the computational graph and chain rule.
Multiple Choice: If ∂f/∂x > 0 at a point, increasing x alone will: (a) decrease f, (b) increase f, (c) leave f unchanged, (d) cannot determine. Answer: (b) increase f locally.
Short Answer: Entry [J]_i,j of the Jacobian equals what? Answer: ∂f_i/∂x_j.

Key Takeaways

ML loss functions are multivariable: training asks how loss changes when one parameter moves and all others stay fixed.
∂f/∂x is defined by the same limit as an ordinary derivative, but with other variables treated as constants.
Geometrically, a partial derivative is the slope of a surface slice along one coordinate axis.
Each weight θ_j in a network contributes one partial derivative ∂L/∂θ_j to the training signal.
The Jacobian matrix organizes all first-order partials of a vector-valued function; for scalar loss, its transpose is the gradient.
Frameworks like PyTorch store per-parameter partials in .grad after reverse-mode differentiation.
Partial derivatives are necessary but not sufficient for steepest-ascent direction—that requires the full gradient (next lectures).
From here, the Chain Rule shows how to compute partials through composed functions—the engine of backpropagation.