Eigenvectors

How This Lesson Fits the Module

The previous lecture, Eigenvalues, answered how much a linear transformation scales certain directions—the scalars λ that satisfy det(A − λI) = 0. This lecture answers the companion question: which directions are scaled, and what structure do they reveal?

Eigenvectors are the special directions a matrix preserves. Together, eigenvalues and eigenvectors decompose complex linear maps into stretch-and-rotate behavior along independent axes—the geometric language behind spectral methods, stability analysis, and dimensionality reduction. This is the capstone of Module 2.1: you began with vectors and matrices, learned how linear maps act through matrix multiplication, and now close the module by understanding the intrinsic axes of any square transformation.

Learning Objectives

By the end of this lesson, students should be able to:

Define eigenvectors and the eigenvalue equation Av = λv, including the requirement that v ≠ 0.
Explain the geometric meaning: eigenvectors are directions preserved (only scaled) under a linear transformation.
Compute eigenvectors for small matrices by solving (A − λI)v = 0 after finding eigenvalues.
Define eigenspaces and determine their dimension from algebraic and geometric multiplicity.
Describe diagonalization as changing basis so a matrix acts by independent scalings along coordinate axes.
Derive the intuition that PCA finds eigenvectors of the covariance matrix and ranks them by eigenvalue magnitude.
Connect eigenstructure to applications in stability, graph algorithms, and machine learning pipelines.
Recognize when a matrix is not diagonalizable and what that implies geometrically.

Introduction: Directions That Survive Transformation

Most vectors change direction when multiplied by a matrix. Rotate the plane by 45° and almost no arrow points the same way afterward. Apply a shear map and every direction tilts—except one.

Yet some transformations leave certain directions untouched. A scaling matrix doubles every vector along the x-axis and triples every vector along the y-axis. Those axes are eigenvectors: the matrix does not rotate them; it only stretches or flips them.

This is not a niche curiosity. In data science, the eigenvectors of a covariance matrix identify the directions of maximum variance—the foundation of Principal Component Analysis (PCA). In dynamical systems, eigenvalues of a Jacobian determine whether perturbations grow or decay. In graph learning, the spectrum of a Laplacian encodes community structure. Engineers who understand eigenvectors understand how linear systems behave in their natural coordinates.

Definition and the Eigenvalue Equation

Definition — Eigenvector and Eigenvalue

Let A be an n × n matrix. A nonzero vector v is an eigenvector of A if there exists a scalar λ such that:

Av = λv

The scalar λ is the eigenvalue associated with v. Equivalently, v lies in the null space of (A − λI) for some eigenvalue λ.

Read the equation carefully. Matrix multiplication normally mixes components: output direction depends on input direction in a coupled way. An eigenvector breaks that coupling. The output is parallel to the input—a pure scaling, possibly with sign flip if λ < 0.

Example — A 2 × 2 Scaling Matrix

Let A = diag(3, −1). Then:

v₁ = [1, 0]^T gives Av₁ = 3v₁, so λ₁ = 3.
v₂ = [0, 1]^T gives Av₂ = −v₂, so λ₂ = −1.

The standard basis vectors are eigenvectors. Every vector in the plane is a linear combination of eigenvectors, so the transformation is fully described by two independent scalings along perpendicular axes.

PrerequisiteThe eigenvalues in this example come from solving det(A − λI) = 0, covered in Eigenvalues. This lecture focuses on the vectors that satisfy the equation once λ is known.

Geometric Meaning: Direction Preserved

The defining property of an eigenvector is geometric, not algebraic: direction is preserved (up to reversal when λ < 0). Length may change; orientation along the line does not.

General Vector

Av typically points in a new direction
Rotation, shear, and scaling mix together
Hard to predict output direction from input alone
Requires full matrix to describe behavior

Eigenvector

Av is parallel to v
Only magnitude (and possibly sign) changes
Behavior reduces to multiplication by scalar λ
Defines an invariant line through the origin

How Eigenvalues Control the Geometry

The eigenvalue λ tells you what happens along the eigenvector line:

Condition on λ	Geometric Effect Along Eigenvector	Intuition
\|λ\| > 1	Expansion	Vectors on that line grow in length
0 < \|λ\| < 1	Contraction	Vectors on that line shrink toward the origin
λ < 0	Reflection + scaling	Direction reverses; \|λ\| still controls magnitude
λ = 0	Collapse to origin	The line maps to a single point

Example — Rotation Has No Real Eigenvectors

A 90° rotation matrix in ℝ² has no real eigenvectors: every nonzero vector changes direction. Its eigenvalues are complex (±i). This is a crucial lesson—eigenvectors are not guaranteed to exist over the reals, even when the matrix is perfectly well-behaved geometrically.

Computing Eigenvectors

Given eigenvalue λ, eigenvectors satisfy the homogeneous system:

(A − λI)v = 0

Solve by row reduction on (A − λI). The free variables parameterize the entire eigenspace. Any nonzero scalar multiple of an eigenvector is also an eigenvector with the same eigenvalue.

Worked Example — Finding Eigenvectors

For A = [[4, 1], [2, 3]], the characteristic polynomial gives λ₁ = 5 and λ₂ = 2.

For λ = 5: (A − 5I) = [[−1, 1], [2, −2]] has rank 1. Solving gives v proportional to [1, 1]^T.

For λ = 2: (A − 2I) = [[2, 1], [2, 1]] gives v proportional to [1, −2]^T.

Geometrically, A stretches the line spanned by [1, 1] by factor 5 and the line spanned by [1, −2] by factor 2. Every vector decomposes into motion along these two independent directions.

Eigenspaces

Definition — Eigenspace

The eigenspace (or eigen-subspace) corresponding to eigenvalue λ is the set of all eigenvectors with that eigenvalue, together with the zero vector:

E_λ = { v ∈ ℝⁿ : Av = λv }

E_λ is a subspace of ℝⁿ. Its dimension is the geometric multiplicity of λ.

Eigenspaces organize eigenvectors into vector spaces rather than isolated arrows. If two eigenvectors share the same eigenvalue, every linear combination of them is also an eigenvector with that eigenvalue. You choose a basis for E_λ to describe the direction(s) of invariance cleanly.

Algebraic vs Geometric Multiplicity

From the characteristic polynomial, each eigenvalue λ has an algebraic multiplicity—how many times ( λ − λ_i) appears as a factor. The geometric multiplicity is dim(E_λ).

A fundamental inequality always holds:

1 ≤ geometric multiplicity ≤ algebraic multiplicity

Common Misconception: “A repeated eigenvalue always means multiple independent eigenvector directions.”

Reality: When geometric multiplicity is less than algebraic multiplicity, the eigenspace is smaller than expected. A single direction may be the only invariant line for that eigenvalue—the transformation can “shear” along that line rather than simply scaling a full subspace.

Example — Defective Matrix

A = [[2, 1], [0, 2]] has characteristic polynomial (2 − λ)², so λ = 2 has algebraic multiplicity 2. But (A − 2I) = [[0, 1], [0, 0]] has rank 1, so the eigenspace is one-dimensional (spanned by [1, 0]^T). This matrix is not diagonalizable: one eigenvalue, but only one independent invariant direction.

Orthogonal Eigenspaces (Symmetric Matrices)

When A is symmetric (A = A^T), a remarkable theorem applies: all eigenvalues are real, and eigenvectors corresponding to distinct eigenvalues are orthogonal. Eigenspaces for different eigenvalues are mutually perpendicular.

This is the mathematical reason PCA axes are uncorrelated principal directions. The covariance matrix is symmetric; its eigenvectors form an orthonormal basis aligned with variance structure in the data.

Diagonalization: The Intuition

Diagonalization is the process of rewriting a matrix so that, in a new coordinate system, it acts as independent scalings along each axis—nothing but eigenvalues on the diagonal.

Definition — Diagonalization

A matrix A is diagonalizable if there exists an invertible matrix P and diagonal matrix D such that:

A = PDP⁻¹

Columns of P are eigenvectors; diagonal entries of D are the corresponding eigenvalues.

What the Formula Means Geometrically

Think of P⁻¹ as change of basis: express any vector in the eigenvector coordinate system. Multiplication by D scales each coordinate independently by the matching eigenvalue. Multiplication by P maps back to the original coordinates.

1. P⁻¹ — Write vector in eigenvector coordinates ↓ 2. D — Scale each coordinate by its eigenvalue ↓ 3. P — Express result in original coordinates

When diagonalization succeeds, the complex action of A factorizes into a change of basis, simple axis-aligned scaling, and a change back. Powers become trivial: A^k = PD^kP⁻¹, because diagonal matrices multiply component-wise.

Engineering Principle

Diagonalization finds the natural coordinates of a linear system. In those coordinates, coupling disappears. This is why spectral decomposition appears in solving recurrence relations, computing matrix exponentials e^At, and analyzing Markov chain steady states.

When Diagonalization Fails

A is diagonalizable over ℝ if and only if there exists a basis of eigenvectors—equivalently, geometric multiplicities sum to n. Defective matrices lack enough eigenvector directions; Jordan normal form (beyond this module) handles the remaining structure.

Not every matrix is diagonalizable, but every symmetric matrix is. Every covariance matrix encountered in PCA is symmetric and therefore safely diagonalizable with real eigenvalues.

PCA as Eigen-Decomposition of Covariance

Principal Component Analysis is the flagship machine-learning application of eigenvectors in this curriculum. At its core, PCA is not a mysterious black box—it is eigen-decomposition of the covariance matrix, ranked by eigenvalue size.

Setup: Centered Data Matrix

Given data points x₁, …, x_m in ℝⁿ, form the centered matrix X whose rows (or columns, depending on convention) are mean-subtracted observations. The covariance matrix is:

C = (1 / (m − 1)) X^TX

C is symmetric and positive semi-definite. Entry C_ij measures how features i and j vary together across the dataset.

The Spectral Theorem in Action

Because C is symmetric, there exists an orthonormal basis of eigenvectors:

Cv_k = λ_kv_k, v_i^Tv_j = 0 for i ≠ j

Each eigenvector v_k is a principal component direction. The eigenvalue λ_k equals the variance of the data projected onto that direction. Larger λ means more spread—more information captured by projecting onto v_k.

Definition — Principal Component

The k-th principal component is the eigenvector of the covariance matrix corresponding to the k-th largest eigenvalue. Projecting data onto the top r principal components yields the best rank-r linear approximation of the data in the least-squares sense.

1. Center data — Subtract feature means ↓ 2. Compute covariance C — Symmetric matrix of feature co-variation ↓ 3. Eigendecompose C — Find eigenvalues λ₁ ≥ λ₂ ≥ … and eigenvectors v_k ↓ 4. Project — Represent each point by coordinates in the top-r eigenvector basis ↓ 5. Reconstruct (optional) — Approximate original data using only top components

Example — Two-Dimensional Cloud of Points

Suppose height and weight measurements form an elongated ellipse tilted off the coordinate axes. The first principal component v₁ points along the long axis of the ellipse—the direction of greatest spread. The second component v₂ is perpendicular and captures residual variation. If λ₂ is much smaller than λ₁, projecting onto v₁ alone loses little information while reducing dimension from 2 to 1.

Connection to Unsupervised LearningPCA is the canonical dimensionality reduction method introduced in Unsupervised Learning. Eigenvectors supply the mathematical mechanism behind that lesson’s intuitive description of “compressing data while preserving structure.”

Why Covariance, Not the Raw Data Matrix?

One can also perform PCA via the singular value decomposition (SVD) of X. For centered data, the left singular vectors of X relate directly to eigenvectors of C. The covariance formulation makes the variance interpretation explicit: each eigenvalue is literally the variance along its eigenvector direction.

Common Mistake

Applying PCA without centering the data. If features are not zero-mean, the first principal component may point toward the mean offset rather than the direction of maximum spread. Always center (and often standardize) before eigendecomposition in practice.

Eigenvectors Across the AI Stack

PCA is the most direct bridge to machine learning, but eigenvectors appear wherever linear structure matters.

Domain	Matrix	What Eigenvectors Reveal
Dimensionality reduction	Covariance matrix	Principal directions of variance (PCA)
Graph neural networks	Graph Laplacian	Cluster structure, spectral embeddings
Dynamical systems / RNNs	Transition or Jacobian matrix	Modes that grow, decay, or oscillate
PageRank / recommendation	Stochastic adjacency matrix	Stationary distribution (eigenvector with λ = 1)
Quantum-inspired methods	Hermitian operators	Observable states with definite measurement values

The pattern is universal: find directions the system treats uniformly. Eigenvectors are those directions; eigenvalues quantify the uniform treatment.

Module 2.1 Recap: From Scalars to Spectra

This lecture closes the linear algebra arc you built across Module 2.1:

Scalars, Vectors, Matrices, Tensors — the objects of computation.
Matrix Multiplication — how linear transformations compose and act.
Dot Product and Cross Product — geometry, projections, and orthogonality.
Eigenvalues and Eigenvectors — the intrinsic scalings and directions of square transformations.

You now have the vocabulary to read loss surfaces, optimization dynamics, and neural network behavior through a linear lens—even when the full models are nonlinear. Local Jacobians are matrices; their eigenvalues predict stability. Hessians are symmetric; their eigenvalues classify critical points. Covariance of activations drives whitening and normalization strategies.

Bridge to Module 2.2

Linear algebra describes static structure and instantaneous maps. Machine learning optimizes functions over time. Module 2.2 — Calculus — supplies derivatives, gradients, and the chain rule that power gradient descent. Eigenvalues told you how a matrix scales fixed directions; gradients will tell you which direction to move to reduce error fastest.

Knowledge Check

Short Answer: State the eigenvector equation and what it means geometrically. Answer: Av = λv; A scales v without changing its direction.
Short Answer: For eigenvalues 4 and −0.5, what happens along each eigenvector under repeated multiplication by A? Answer: The λ=4 direction grows; the λ=−0.5 direction shrinks and alternates sign.
Short Answer: What is the eigenspace for λ=3 if (A − 3I) has rank 2 and A is 3×3? Answer: A 1-dimensional line (nullity = 3 − 2 = 1).
Short Answer: Why must the covariance matrix be symmetric, and what does that imply for PCA eigenvectors? Answer: Cov(X_i,X_j)=Cov(X_j,X_i); real symmetric matrices have real orthogonal eigenvectors.
Short Answer: Why does diagonalization require a full basis of eigenvectors? Answer: P must be invertible, so its columns (eigenvectors) must span R^n.
Short Answer: PCA reports PC1 explains 95% of variance. What does that mean in eigenvalues? Answer: λ_1 / sum(λ) ≈ 0.95.
Short Answer: Why can a 2D rotation lack real eigenvectors, and where do complex eigenvalues still matter? Answer: No real direction is merely scaled; complex λ encode oscillation/stability (e.g. dynamical systems).
Short Answer: Why must an eigenvector be nonzero? Answer: The zero vector satisfies A0 = λ0 for every λ, so it carries no directional information.
True/False: A real symmetric matrix has real eigenvalues and orthogonal eigenvectors. Answer: True—this is why PCA axes are orthogonal.
Multiple Choice: PCA eigendecomposes which matrix? (a) the Jacobian, (b) the covariance matrix, (c) the identity, (d) a rotation matrix. Answer: (b).

Key Takeaways

An eigenvector v satisfies Av = λv; the matrix preserves its direction and scales it by λ.
Eigenspaces collect all eigenvectors for a given λ; their dimension is geometric multiplicity.
Diagonalization writes A = PDP⁻¹, turning matrix action into independent scalings in an eigenvector basis.
PCA eigendecomposes the covariance matrix; eigenvalues rank variance along orthogonal principal directions.
Symmetric matrices have real eigenvalues and orthogonal eigenvectors—the theoretical backbone of PCA.
Eigenstructure connects linear algebra to unsupervised learning, graph methods, and stability analysis throughout AI.
Module 2.1 ends here; Module 2.2 extends the toolkit with calculus for optimization.