Derivatives

How This Lesson Fits the Module

Module 2.1: Linear Algebra gave you the static language of AI—vectors, matrices, and eigenstructure that describe how data is represented and how linear maps act. The capstone lecture, Eigenvectors, closed that arc by identifying the intrinsic directions a transformation preserves. Machine learning, however, does not only represent data; it improves models by adjusting parameters over time.

Derivatives are the first tool in Module 2.2: Calculus for that dynamic story. A derivative measures how fast a quantity changes—the slope of a curve at a single point. In AI, that quantity is often a loss function: error as a function of weights. The derivative tells the optimizer which way to nudge each parameter to reduce error. This lecture builds the single-variable foundation; later lectures extend it to partial derivatives, the chain rule, gradients, and gradient descent.

If linear algebra describes the landscape, calculus tells you how to move across it.

Learning Objectives

By the end of this lesson, students should be able to:

Interpret a derivative as an instantaneous rate of change and as the slope of the tangent line to a curve.
Read and write derivative notation: f′(x), dy/dx, and Leibniz-style differentials.
Apply the power rule and common differentiation rules to elementary functions.
Connect the shape of a loss curve to the sign and magnitude of its derivative at a point.
Relate derivatives to the optimization narrative introduced in Module 2.1—from eigenstructure of quadratic forms to slope along a training curve.
Explain why automatic differentiation (e.g., PyTorch autograd) is central to neural network training.
Distinguish average rate of change from instantaneous rate of change.
Recognize common misconceptions about derivatives, differentiability, and zero slope.

Introduction: From Static Structure to Motion

In Machine Learning, you learned that training improves a model’s performance on an objective. That improvement is not magic—it is a sequence of small, informed adjustments to parameters. Each adjustment asks the same question: if I change this weight slightly, does the loss go up or down, and by how much?

That question is answered by a derivative. Whether you are fitting a line to housing data or training a billion-parameter language model, the core idea is identical: measure sensitivity of output to input, then move in the direction that reduces error.

Module 2.1 equipped you to think in high dimensions—gradients are vectors of partial derivatives, and eigenvectors of a Hessian classify curvature at critical points. This lecture starts one dimension lower, where the geometry is easy to draw and the algebra is easy to verify by hand. Master the 1D case here; everything in Module 2.2 generalizes from it.

Rate of Change: Average vs Instantaneous

Suppose a model’s loss L depends on a single weight w. After training step k, the weight is w_k and loss is L(w_k). After step k + 1, the weight moves to w_k+1. The average rate of change of loss over that interval is:

ΔL / Δw = (L(w_k+1) − L(w_k)) / (w_k+1 − w_k)

This is the slope of a secant line—the straight line through two points on the loss curve. It tells you how loss changed over a finite step. Optimizers care about something finer: behavior in the limit as the step shrinks to zero.

Definition — Derivative

Let f be a function of a real variable x. The derivative of f at x = a, written f′(a), is the instantaneous rate of change of f at a:

f′(a) = lim_h→0 [f(a + h) − f(a)] / h

When this limit exists, f is differentiable at a. Geometrically, f′(a) is the slope of the tangent line to the graph of f at the point (a, f(a)).

Average Rate of Change

Uses two distinct points on the curve
Slope of the secant line
Depends on the size of the interval Δx
What you measure between training steps

Derivative (Instantaneous)

Uses one point; limit as Δx → 0
Slope of the tangent line
Local sensitivity at a single input
What optimizers approximate with small steps

Positive derivative at w₀ means loss increases as w increases locally—so decreasing w should help. Negative derivative means the opposite. Near a minimum, the derivative approaches zero: the tangent line is flat, and the model has found a local resting point (not necessarily the global best).

Notation: f′(x) and dy/dx

Calculus uses several equivalent notations. Fluency with all of them prevents confusion when reading papers, textbooks, and framework documentation.

Notation	Read As	Typical Context
f′(x)	“f prime of x”	Compact; common in proofs and Stewart-style texts
dy/dx	“dee y dee x”	Leibniz; emphasizes y as a function of x
d/dx [f(x)]	“derivative of f with respect to x”	Operator form; extends cleanly to partial derivatives
f′′(x), d²y/dx²	Second derivative	Curvature; later connects to convexity and Hessians

If y = f(x), then dy/dx and f′(x) denote the same object. In ML code, you will see yet another style—tensor.grad in PyTorch—but the mathematical meaning remains: sensitivity of output to input.

Bridge from Linear AlgebraIn Module 2.1, Eigenvectors identified directions preserved under a matrix map. For a quadratic loss L(w) = w^TAw with symmetric positive definite A, the eigenvectors of A are the principal axes of the bowl-shaped surface. Along each axis, the loss is a 1D parabola—and its derivative is a linear function of w. Eigenstructure tells you which directions decouple; derivatives tell you how fast loss changes along each.

Computing Derivatives: The Power Rule

Evaluating the limit definition by hand for every function is impractical. Rules reduce differentiation to algebra. The most frequently used rule in introductory calculus—and in the polynomial pieces of neural network activations—is the power rule.

Rule — Power Rule

If f(x) = xⁿ where n is any real exponent (in the cases we use most often, a positive integer), then:

f′(x) = n xⁿ⁻¹

Example — Power Rule in Action

d/dx [x³] = 3x²
d/dx [x⁻¹] = −x⁻²
d/dx [√x] = d/dx [x^1/2] = (1/2)x^−1/2
d/dw [5w² + 3w] = 10w + 3 — a linear loss gradient in one weight

Common Differentiation Rules

Neural networks compose simple functions—affine maps, powers, exponentials, logarithms. The table below collects the rules you will use repeatedly before the Chain Rule lecture automates composition.

Rule	Formula	AI Relevance
Constant	d/dx [c] = 0	Bias terms vanish from derivatives of other variables
Constant Multiple	d/dx [c f(x)] = c f′(x)	Learning rate scales gradient contributions
Sum / Difference	d/dx [f ± g] = f′ ± g′	Total loss is often a sum of per-example losses
Product	d/dx [f g] = f′g + fg′	Needed before chain rule for factored expressions
Quotient	d/dx [f/g] = (f′g − fg′) / g²	Ratio losses and normalized terms
Exponential	d/dx [e^x] = e^x	Softmax and cross-entropy derivations
Natural Log	d/dx [ln x] = 1/x	Log-likelihood objectives; numerically stable loss forms

Engineering Principle

You rarely differentiate by hand in production, but you must recognize which rules a framework applies. When a gradient looks wrong, the bug is often a misunderstood composition, a forgotten product term, or a discontinuity—not a faulty GPU.

Loss Curves: Reading the Derivative

Training a model means tracing a path through parameter space that lowers loss. In one dimension, that path is a walk along a curve. The derivative is your local compass.

Loss Curve Feature	Derivative Sign / Size	Optimizer Intuition
Steep uphill (left to right)	Large positive L′(w)	Decrease w aggressively
Steep downhill	Large negative L′(w)	Increase w aggressively
Flat near a minimum	L′(w) ≈ 0	Stop or use tiny steps; watch for saddle vs minimum
Gentle slope	Small \|L′(w)\|	Slow progress; may need adaptive learning rate

Consider mean squared error for one weight: L(w) = (y − wx)². Expanding gives a quadratic in w—a parabola. Its derivative L′(w) = −2x(y − wx) is linear in w, zero at the best-fit weight w* = y/x (when x ≠ 0). This is the simplest picture of gradient-based fitting: follow the slope until it vanishes.

Example — Loss Landscape Slice

Imagine freezing every neural network weight except one bias b. Plot loss vs b. You see a 1D slice through a high-dimensional surface. At each b, the derivative dL/db tells you whether increasing b raises or lowers loss. Training all parameters simultaneously generalizes this idea to many derivatives at once—the subject of Partial Derivatives and Gradient.

Common Misconception: “If the derivative is zero, we have found the best model.”

Reality: f′(x) = 0 identifies a critical point—a candidate for a local minimum, local maximum, or saddle point. In high-dimensional non-convex losses (typical deep networks), most critical points are saddles or shallow local minima. Zero derivative is a necessary condition for a local minimum, not a sufficient guarantee of global optimality.

Common Misconception: “Derivatives are only for smooth textbook functions; ReLU networks are not differentiable.”

Reality: ReLU(x) = max(0, x) is not differentiable at x = 0, but is differentiable everywhere else with derivative 0 or 1. Subgradients and automatic differentiation handle piecewise-smooth functions used in practice. Frameworks compute derivatives at almost every point encountered during training.

Bridge to Neural Network Training

A feedforward layer computes z = Wx + b, then applies an activation. Loss measures deviation from targets. Training adjusts W and b to reduce loss. Each scalar parameter w_ij contributes through a long chain of operations—but locally, the same derivative logic applies: how does loss change if I perturb this one weight?

1. Forward pass — Compute predictions and loss L ↓ 2. Derivatives — Compute ∂L/∂w for each parameter (1D derivatives generalized) ↓ 3. Update — w ← w − η · ∂L/∂w (learning rate η) ↓ 4. Repeat — Until loss stabilizes or validation metric peaks

Manual backpropagation through a deep network is error-prone. Automatic differentiation records operations during the forward pass and applies differentiation rules in reverse. In PyTorch, enabling gradients on a tensor and calling .backward() on the scalar loss invokes autograd—the engine that implements the chain rule across the computation graph. You define the forward computation; the framework supplies derivatives.

Sketch — PyTorch Autograd (Conceptual)

import torch

w = torch.tensor(2.0, requires_grad=True)
loss = (y - w * x) ** 2   # scalar loss, e.g. MSE with one weight

loss.backward()            # autograd computes d(loss)/d(w)
print(w.grad)              # same derivative the power rule gives: -2*x*(y - w*x)

The value in w.grad is the derivative you would compute with the power rule and chain rule—ready to drive a weight update.

Why This Matters for AI Engineers

Every major deep learning framework—PyTorch, TensorFlow, JAX—is built on automatic differentiation. You do not implement backprop by hand for each architecture. You do need to know what a derivative means, when it is zero, why it can vanish (vanishing gradients), and how loss curvature affects training stability. Module 2.1’s eigenvalues of a Hessian quantify second-order curvature; this lecture’s first-order derivative is the workhorse that actually moves parameters every step.

Module 2.1 Recap: Where Calculus Picks Up

Before diving deeper into Module 2.2, anchor the new material to what you already built:

Vectors and Matrices — parameters and data live in structured numeric form.
Matrix Multiplication — forward passes are sequences of linear maps.
Eigenvalues and Eigenvectors — intrinsic directions and scalings; PCA and curvature analysis.

Linear algebra told you how transformations act on space. Calculus tells you how to improve the parameters defining those transformations when a loss function judges their quality.

Knowledge Check

Short Answer: In one sentence, what does f′(a) measure? Answer: The instantaneous rate of change of f at x = a, equivalently the slope of the tangent line at that point.
Computation: Find d/dx [4x⁵ − 3x² + 7]. Answer: 20x⁴ − 6x.
True/False: If L′(w) > 0 at a point, increasing w locally increases loss. Answer: True.
Multiple Choice: The derivative of e^x is: (a) xe^x−1, (b) e^x, (c) ln x, (d) 1. Answer: (b).
Short Answer: What is the difference between a secant slope and a tangent slope? Answer: Secant uses two points (average rate); tangent uses one point as a limit (instantaneous rate).
Computation: For L(w) = (y − w)², find L′(w) and the value of w where it equals zero. Answer: L′(w) = −2(y − w); zero at w = y.
Short Answer: How do eigenvectors of a symmetric Hessian relate to directions of a loss surface? Answer: They give orthogonal principal directions; eigenvalues indicate curvature (steepness) along each.
True/False: PyTorch autograd eliminates the need to understand what a derivative means. Answer: False — autograd computes derivatives; engineers still need intuition for signs, magnitudes, and failure modes.
Multiple Choice: At a local minimum of a differentiable function, the derivative is: (a) undefined, (b) positive, (c) negative, (d) zero. Answer: (d).
Short Answer: Why is the power rule sufficient for differentiating any polynomial? Answer: Each term is a constant times a power of x; sum and constant-multiple rules apply term by term.

Key Takeaways

A derivative is an instantaneous rate of change—the slope of the tangent line—defined as a limit of average rates.
Notation f′(x) and dy/dx are interchangeable when y = f(x).
The power rule and common rules reduce differentiation of elementary functions to routine algebra.
On a loss curve, the sign of the derivative indicates the direction to adjust a weight; magnitude indicates urgency.
Zero derivative marks a critical point, not necessarily the global best solution.
Module 2.1 eigenstructure describes curvature directions; first derivatives drive the actual parameter updates in training.
Neural network training relies on automatic differentiation (PyTorch autograd) to compute derivatives across composed operations.
Next: Partial Derivatives extend this 1D story to the many parameters of real models.