Eigenvalues

How This Lesson Fits the Module

The Cross Product lecture completed the geometric toolkit for vectors in three dimensions—orthogonal directions, area, and torque-like interpretations. Those operations describe what happens to individual vectors under specific constructions.

Eigenvalues shift the question: when a square matrix A transforms an entire space, are there special directions that the transformation merely stretches or compresses, without rotating them to a different line? The scalar factors in that answer—the stretching rates—are eigenvalues. They encode how a linear map amplifies or damps signal along privileged axes, and they appear everywhere in machine learning: principal component analysis, stability of recurrent dynamics, and the spectral behavior of weight matrices during training.

This lecture defines eigenvalues, develops intuition for the characteristic equation, connects eigenvalues to PCA, and previews why their magnitude matters for neural network stability. The companion lecture on Eigenvectors completes the picture by focusing on the directions themselves.

Learning Objectives

By the end of this lesson, students should be able to:

State the eigenvalue equation Av = λv and explain what each symbol represents.
Interpret an eigenvalue geometrically as a stretch factor along an invariant direction.
Derive and explain the characteristic equation det(A − λI) = 0 at an intuitive level.
Compute eigenvalues for small matrices (2×2 and simple 3×3 cases).
Relate eigenvalues of a covariance matrix to variance explained in PCA.
Describe, at a high level, how eigenvalue magnitudes connect to stability and gradient behavior in neural networks.
Distinguish eigenvalues from eigenvectors and know which lecture covers each.

Introduction: Stretching Without Turning

Matrix multiplication is a linear transformation: it can rotate, shear, reflect, and scale space. For a general matrix A, most input vectors v emerge pointing in a different direction after multiplication. Eigenvalues identify the exceptions—directions where the output is parallel to the input.

Consider a 2×2 matrix that doubles every horizontal component but leaves vertical components unchanged. The vector (1, 0)^T becomes (2, 0)^T—same direction, twice the length. The vector (0, 1)^T is unchanged. Those special scale factors—2 and 1—are eigenvalues. The directions (1, 0) and (0, 1) are eigenvectors (covered in depth in the next lecture).

Why eigenvalues matter for AI practitioners:

Dimensionality reduction — PCA ranks directions by their eigenvalues on the covariance matrix.
Dynamics and stability — Repeated application of a matrix (as in recurrent networks) grows or shrinks signals according to eigenvalue magnitudes.
Understanding transformations — Eigenvalues summarize how a learned linear layer amplifies or suppresses different modes of variation in data.

The Eigenvalue Equation

Definition — Eigenvalue

Let A be an n×n matrix with entries in ℝ (or ℂ). A scalar λ is an eigenvalue of A if there exists a nonzero vector v ∈ ℝⁿ such that

Av = λv

Any nonzero v satisfying this equation is called an eigenvector corresponding to λ. The pair (λ, v) is an eigenpair. The zero vector is excluded because it satisfies the equation for every λ and carries no directional information.

Read the equation operationally:

A acts on v by matrix–vector multiplication.
λv is the same vector scaled by the scalar λ.
Equality means the transformation does not rotate v off its span; it only rescales it (or reflects it, if λ < 0).

Input eigenvector v (a direction in space) ↓ Apply linear map: Av ↓ Result lies on the same line as v ↓ Scale factor λ tells how much length changed (sign included)

Symbol	Meaning	Notes
A	Square matrix (n×n)	Defines the linear transformation; eigenvalues are properties of A, not of individual vectors
v	Eigenvector	Must be nonzero; only direction matters (scaling v by a constant gives another eigenvector for the same λ)
λ	Eigenvalue	Can be real or complex; \|λ\| is the magnitude of stretch when λ is real and positive
I	Identity matrix	Used to rewrite the equation as (A − λI)v = 0

Prerequisite CheckThis lecture assumes comfort with matrix–vector multiplication from Matrix Multiplication and square matrices from Matrices. Determinants are used below for intuition; a full determinant treatment may appear elsewhere in the curriculum.

From Av = λv to the Characteristic Equation

Rearrange the eigenvalue equation:

Av − λv = 0 ⇒ (A − λI)v = 0

This is a homogeneous linear system. We want a nontrivial solution v ≠ 0. For square (A − λI), nontrivial solutions exist precisely when the matrix is singular—that is, when its determinant vanishes.

Definition — Characteristic Equation

The characteristic polynomial of A is

p(λ) = det(A − λI)

The characteristic equation is p(λ) = 0. The roots of this polynomial are exactly the eigenvalues of A.

Why This Makes Sense

Think of λ as a tuning parameter. As λ varies, the matrix (A − λI) shifts between invertible and non-invertible. Eigenvalues are the values of λ where the shift is exactly right to collapse a direction to zero under (A − λI), leaving a surviving direction v that A scales by λ.

Degree — For an n×n matrix, det(A − λI) is a degree-n polynomial in λ. Hence there are at most n eigenvalues in ℂ (counting algebraic multiplicity).
Trace and determinant — For a 2×2 matrix with eigenvalues λ₁, λ₂: tr(A) = λ₁ + λ₂ and det(A) = λ₁λ₂. These identities generalize and are useful for quick checks.
Triangular matrices — If A is upper or lower triangular, eigenvalues are the diagonal entries. No characteristic polynomial expansion required.

Example — 2×2 Eigenvalues

Let A = [[2, 1], [0, 3]] (upper triangular).

Characteristic polynomial:

det([[2−λ, 1], [0, 3−λ]]) = (2 − λ)(3 − λ) − 0 = λ² − 5λ + 6

Solve: λ² − 5λ + 6 = 0 ⇒ (λ − 2)(λ − 3) = 0, so λ₁ = 2 and λ₂ = 3.

Check: tr(A) = 2 + 3 = 5 = λ₁ + λ₂. det(A) = 6 = λ₁λ₂.

Interpretation: One eigen-direction stretches by 2; another by 3. Because the matrix is triangular, those directions align with the coordinate axes.

Example — Symmetric 2×2 Matrix

Let A = [[4, 2], [2, 1]].

det([[4−λ, 2], [2, 1−λ]]) = (4−λ)(1−λ) − 4 = λ² − 5λ = λ(λ − 5)

Eigenvalues: λ₁ = 0 and λ₂ = 5.

The zero eigenvalue means A collapses some nonzero direction to the origin—A is singular (det = 0). The positive eigenvalue 5 identifies the direction of maximum stretch. Symmetric matrices always have real eigenvalues—a fact that underpins PCA on covariance matrices.

Algebraic vs. Geometric Multiplicity

The algebraic multiplicity of λ is its multiplicity as a root of p(λ). The geometric multiplicity is the dimension of the eigenspace { v : Av = λv }. Geometric multiplicity never exceeds algebraic multiplicity. When they differ, A is defective and cannot be diagonalized; this distinction matters in advanced stability analysis but is secondary at introductory level.

Geometric and Spectral Intuition

Eigenvalues are not merely roots of a polynomial—they are summaries of how a transformation behaves along its principal axes.

|λ| > 1

Repeated application of A along that eigen-direction amplifies components. In dynamics, modes grow exponentially.

|λ| < 1

Components along that direction decay under repeated multiplication. The mode is damped.

λ < 0

Each application flips orientation (reflection) while scaling by |λ|. Oscillatory behavior can appear when combined with complex eigenvalues in higher dimensions.

λ = 0

A is singular. Some direction is collapsed to zero. Common in rank-deficient data covariance matrices after centering.

For a symmetric matrix S, all eigenvalues are real and eigenvectors corresponding to distinct eigenvalues are orthogonal. For orthogonal matrices Q (rotations and reflections), every eigenvalue satisfies |λ| = 1: lengths are preserved, only directions may change. These special cases appear constantly in machine learning theory and practice.

Connection to Principal Component Analysis (PCA)

PCA is the most direct machine-learning application of eigenvalues. Given a centered data matrix X (each column or row a sample; each feature zero-mean), the covariance matrix is

Σ = (1/n) X^TX (or equivalent normalization)

Σ is symmetric and positive semidefinite. Its eigenvalues λ₁ ≥ λ₂ ≥ … ≥ λ_d ≥ 0 measure variance along the corresponding eigenvector directions.

Centered data X ↓ Covariance matrix Σ = X^TX / n ↓ Solve det(Σ − λI) = 0 ↓ Sort eigenvalues λ₁ ≥ λ₂ ≥ … ↓ Keep top-k directions for reduced representation

Eigenvalue Role in PCA	Interpretation
Largest λ₁	Direction of maximum variance—the first principal component captures the most spread in the data
Sum of eigenvalues	Equals tr(Σ), the total variance (for trace-normalized covariance)
λ_i / ∑_j λ_j	Fraction of variance explained by the i-th principal component; used to choose k
Near-zero eigenvalues	Indicate near-collinearity or redundant features; data lies close to a lower-dimensional subspace

Engineering Note — Eigendecomposition vs. SVD

Production PCA often uses the singular value decomposition (SVD) of X rather than explicitly forming Σ. For symmetric Σ, singular values σ_i relate to eigenvalues by σ_i² = λ_i. The eigenvalue viewpoint explains what PCA optimizes (variance along orthogonal directions); SVD explains how large-scale libraries compute it stably.

Example — Variance Explained

Suppose eigenvalues of a 4-dimensional covariance matrix are λ = [8.0, 2.0, 0.8, 0.2]. Total variance = 11.0.

PC1 explains 8.0/11.0 ≈ 72.7% of variance.
PC1 + PC2 explain (8.0 + 2.0)/11.0 ≈ 90.9%.
Keeping two components retains most signal for visualization or compression.

Eigenvalues and Stability in Neural Networks

Neural networks are nonlinear, but eigenvalue reasoning still illuminates training dynamics—especially where linear approximations dominate: initialization, recurrent unfolding, and Jacobian analysis near fixed points.

Linear Recurrence and Spectral Radius

If a hidden state evolves as h_t+1 = Wh_t (a linear recurrent model), then h_t = W^th₀. Behavior after many steps is governed by the eigenvalue of W with largest magnitude—the spectral radius ρ(W) = max_i |λ_i|.

ρ(W) < 1 — All modes decay; the origin is stable. Gradients through long unrolls do not explode in this linear model.
ρ(W) > 1 — At least one mode grows exponentially; activations and gradients can explode across time steps.
ρ(W) = 1 — Marginal stability; small nonlinearities or noise determine long-term behavior.

Vanishing and exploding gradients in RNNs are, in part, spectral phenomena: repeated multiplication by W scales error signals by powers of eigenvalues. LSTM and GRU gates were designed partly to control effective eigenvalue behavior over long sequences.

Initialization and Layer Scaling

At initialization, deep feedforward networks approximate compositions of random linear maps. If weight matrices have spectral radius much larger than 1, forward activations grow layer by layer; much smaller than 1, they shrink. Xavier and He initialization schemes set variance scales so that typical singular value spectra keep signal variance in a reasonable range through depth—implicitly managing eigenvalue-like growth without computing them explicitly at every step.

Jacobian Eigenvalues (Local Stability)

For a nonlinear update h_t+1 = f(h_t), linearization near a fixed point h* gives Jacobian J = ∂f/∂h evaluated at h*. Local stability depends on whether eigenvalues of J lie inside the unit circle (discrete time) or have negative real parts (continuous time). This is advanced relative to introductory linear algebra, but the headline is clear: eigenvalue magnitude tells you whether small perturbations grow or shrink.

High-Level ML Summary

Students need not compute Jacobians by hand in introductory courses. They should remember three slogans:

PCA: big eigenvalues of covariance = important directions.
RNNs: eigenvalues of recurrent weights near 1 cause long-memory or instability.
Depth: poorly scaled weights act like repeated multiplication by matrices with extreme eigenvalues—signals vanish or explode.

Computing Eigenvalues in Practice

For n ≤ 4, the characteristic polynomial is manageable by hand. For large n, explicit polynomial root-finding is numerically unstable. Production libraries use iterative methods:

Power iteration — Finds the dominant eigenvalue/eigenvector by repeated multiplication.
QR algorithm — Standard dense eigenvalue solver for general matrices.
Lanczos / Arnoldi — Krylov subspace methods for large sparse matrices common in scientific computing.

In Python, numpy.linalg.eig returns eigenvalues and eigenvectors; numpy.linalg.eigvalsh exploits symmetry (as in covariance matrices) for speed and guaranteed real results.

Common Misconceptions

Misconception 1: “Every matrix has real eigenvalues.”

Why people believe it: PCA and covariance examples only show positive real eigenvalues.

Reality: Real nonsymmetric matrices can have complex conjugate eigenvalues (e.g., rotation matrices). Symmetric (and more generally Hermitian) matrices always have real eigenvalues.

Misconception 2: “Eigenvalues and eigenvectors are the same thing.”

Why people believe it: The terms appear together and are computed simultaneously.

Reality: λ is a scalar stretch factor; v is a direction. This lecture focuses on λ; the next lecture, Eigenvectors, develops direction, orthogonality, and bases.

Misconception 3: “If det(A) = 0, then λ = 0 is the only eigenvalue.”

Why people believe it: Zero determinant is conflated with zero eigenvalue.

Reality: det(A) = 0 means at least one eigenvalue is zero (product of eigenvalues is zero). Other eigenvalues may be nonzero.

Misconception 4: “Larger eigenvalues always mean more important features in every ML model.”

Why people believe it: PCA intuition is over-generalized.

Reality: In PCA, large covariance eigenvalues mean high variance—not necessarily predictive power for a target label. Supervised models may weight low-variance directions heavily if they correlate with the outcome.

Quick Knowledge Check

Short Answer: State the eigenvalue equation. Answer: Av = λv for nonzero v
Short Answer: What is the characteristic equation? Answer: det(A − λI) = 0
True/False: A 3×3 matrix can have at most three eigenvalues in ℂ counting multiplicity. Answer: True
Multiple Choice: For upper triangular A, eigenvalues are found on the: Answer: diagonal
Short Answer: If λ = 0 is an eigenvalue, what does that imply about det(A)? Answer: det(A) = 0 (singular matrix)
Short Answer: In PCA, what does the largest eigenvalue of the covariance matrix represent? Answer: Variance along the first principal component (direction of maximum variance)
True/False: Symmetric covariance matrices always have nonnegative eigenvalues. Answer: True (positive semidefinite)
Multiple Choice: In a linear recurrence h_t+1 = Wh_t, long-term growth is dominated by eigenvalues with: Answer: largest magnitude |λ|
Short Answer: If all eigenvalues of W satisfy |λ| < 1, what happens to h_t as t → ∞ in the linear model? Answer: h_t → 0 (stable decay)
True/False: Eigenvectors are fully covered in this lecture. Answer: False — see the Eigenvectors lecture next

Key Takeaways

An eigenvalue λ satisfies Av = λv for some nonzero direction v; it measures stretch (and possibly sign flip) along an invariant axis.
Eigenvalues are roots of the characteristic equation det(A − λI) = 0—a degree-n polynomial for n×n matrices.
For 2×2 matrices, tr(A) equals the sum of eigenvalues and det(A) equals their product.
In PCA, eigenvalues of the covariance matrix rank directions by variance; choosing top-k components is choosing the k largest eigenvalues.
Spectral radius ρ(W) = max |λ_i| governs stability of linear recurrences and informs vanishing/exploding signal behavior in recurrent models.
Weight initialization and normalization in deep networks implicitly manage how eigenvalue-like scaling accumulates across layers.
Eigenvectors—the directions that realize these scalings—are the subject of the next lecture.