Matrices

How This Lesson Fits the Module

The previous lecture on Vectors introduced ordered lists of numbers—quantities with magnitude and direction that live in space. Matrices are the natural next step: rectangular grids of numbers that organize many vectors at once and, more importantly, act on vectors through linear transformations.

Every neural network layer, every batch of training data, and every attention score in a transformer is ultimately stored and computed as matrices. Before you can multiply matrices, compute dot products, or find eigenvalues later in this module, you must be fluent in what a matrix is, how its dimensions are read, and what the basic operations mean.

Learning Objectives

By the end of this lesson, students should be able to:

Define a matrix and state its dimensions as m × n (rows × columns).
Interpret a matrix as a linear transformation that maps vectors to vectors.
Perform matrix addition and scalar multiplication element-wise.
Compute and interpret the transpose of a matrix.
Recognize the identity matrix and explain why it is the “do nothing” transformation.
Connect matrices to AI: weight matrices in neural networks, batched data, and attention mechanisms.
Distinguish common beginner errors involving dimension mismatches and row/column confusion.

Introduction: From Vectors to Grids

A single vector can represent one data point—a word embedding, a pixel color, a patient’s vital signs. Real AI systems rarely process one vector at a time. They process thousands simultaneously, organized in tables, images, sequences, and weight parameters spread across layers.

A matrix is the mathematical object that captures all of this structure. It is a rectangular array of numbers arranged in rows and columns. Engineers use matrices because linear algebra gives us efficient algorithms—especially on GPUs—for the operations that dominate deep learning: multiply, add, transpose, and decompose.

If vectors are the atoms of linear algebra, matrices are the molecules. They combine numbers into structures that both store data and transform it.

What Is a Matrix?

Definition — Matrix

A matrix is a rectangular array of numbers (or other elements) arranged in rows and columns. The entry in row i and column j is denoted a_ij or A_ij.

Consider a matrix A with three rows and two columns:

A = | 1   4 |
    | 2   5 |
    | 3   6 |

Here, a₁₁ = 1, a₁₂ = 4, a₂₁ = 2, and so on. Each row runs horizontally; each column runs vertically. In code, the same matrix might appear as a 2D NumPy array or a PyTorch tensor with shape (3, 2).

Notation Convention

Matrices are usually written with bold uppercase letters (A, W, I). Vectors are bold lowercase (v, x). Scalars are italic lowercase (c, λ). This convention appears throughout machine learning papers and frameworks.

Dimensions: m × n

The dimension (or shape) of a matrix is always reported as rows × columns—never the reverse.

Matrix	Rows (m)	Columns (n)	Dimension
Example A above	3	2	3 × 2
Identity matrix I₃	3	3	3 × 3 (square)
Neural net weight layer (512 inputs → 256 outputs)	256	512	256 × 512
Mini-batch of 32 images, each flattened to 784 pixels	32	784	32 × 784

Special cases worth naming:

Column vector — An n × 1 matrix; a vector written vertically.
Row vector — A 1 × n matrix; a vector written horizontally.
Square matrix — m = n; required for determinants, inverses, and eigenvalues (covered later in this module).

Common Engineering Mistake

Confusing rows and columns when reading shape. In NumPy and PyTorch, shape = (m, n) means m rows and n columns. Saying “784 by 32” when you mean 32 samples of 784 features will cause silent bugs in matrix multiplication—or loud shape errors if you are lucky.

Matrices as Linear Transformations

Beyond storage, a matrix represents a function: it takes a vector as input and produces a new vector as output. When this function is linear—preserving vector addition and scalar multiplication—it can be written as matrix–vector multiplication.

Definition — Linear Transformation

A transformation T is linear if for all vectors u, v and scalar c:
T(u + v) = T(u) + T(v) and T(cu) = c T(u).

Geometrically, a 2 × 2 matrix can rotate, scale, shear, or reflect points in the plane. A 3 × 3 matrix does the same in 3D space. In AI, we rarely visualize these transformations, but the math is identical: each layer of a neural network applies a learned linear transformation (followed by a nonlinearity) to its input vector or batch.

Example — Scaling in 2D

The matrix S = [[2, 0], [0, 3]] doubles the x-coordinate and triples the y-coordinate of every input vector. The point (1, 1) maps to (2, 3). This is a diagonal scaling matrix—one of the simplest linear transformations.

The key compatibility rule: an m × n matrix transforms n-dimensional input vectors into m-dimensional output vectors. Columns of the matrix tell you where the basis vectors land. Matrix–vector multiplication is covered in depth in Matrix Multiplication; for now, remember that shape compatibility is non-negotiable.

Input vector x ∈ ℝⁿ ↓ Multiply by matrix W ∈ ℝ^m×n ↓ Output vector y = Wx ∈ ℝ^m

Matrix Addition

Two matrices can be added only when they have the same dimensions. Addition is performed element-wise: add each corresponding entry.

| 1  2 |   | 5  6 |   | 6   8 |
| 3  4 | + | 7  8 | = | 10  12 |

Properties that mirror ordinary arithmetic:

Commutative: A + B = B + A
Associative: (A + B) + C = A + (B + C)
Zero matrix: A + 0 = A, where 0 has all entries equal to zero

In neural networks, addition appears when combining residual connections (y = F(x) + x), accumulating gradients, and merging bias terms—though biases are often added via broadcasting rather than full matrix addition.

Scalar Multiplication

Multiplying a matrix by a scalar (a single number) multiplies every entry by that scalar. Scalars are the subject of the next lecture; here they act as uniform scaling factors on the entire matrix.

3 × | 1  2 |   | 3   6 |
        | 3  4 | = | 9  12 |

Scalar multiplication distributes over matrix addition: c(A + B) = cA + cB. During training, learning rates scale gradient matrices before they are subtracted from weights—every weight update is touched by scalar multiplication.

Up NextThe next lecture, Scalars, examines scalars as the simplest building block—single numbers that scale vectors and matrices without changing shape.

The Transpose

The transpose of a matrix flips it across its main diagonal: rows become columns and columns become rows. The transpose of A is written A^T or A^⊤.

A = | 1  4  7 |        A^T = | 1  2  3 |
    | 2  5  8 |              | 4  5  6 |
    | 3  6  9 |              | 7  8  9 |

(3 x 3)                    (3 x 3)

If A is m × n, then A^T is n × m. Key properties:

(A^T)^T = A
(cA)^T = cA^T
(A + B)^T = A^T + B^T
(AB)^T = B^TA^T (order reverses under multiplication)

Transposes appear everywhere in AI: converting row vectors to column vectors, forming dot products as a^Tb, and building attention score matrices in transformers. See Dot Product for the geometric meaning behind a^Tb.

The Identity Matrix

Definition — Identity Matrix

The identity matrix I_n is the n × n square matrix with ones on the main diagonal and zeros elsewhere. For any compatible matrix A: AI = A and IA = A.

I_3 = | 1  0  0 |
      | 0  1  0 |
      | 0  0  1 |

Think of I as the “multiply by one” of matrix arithmetic. It represents the transformation that leaves every vector unchanged—the identity function in linear form. In initialization schemes, adding a small multiple of I to weight matrices can stabilize training. In attention mechanisms, masking and softmax normalization interact with identity-like structures to preserve certain dimensions.

Matrices in Artificial Intelligence

Matrices are not abstract homework exercises in AI engineering. They are the data structures that make modern systems possible.

Weight Matrices in Neural Networks

Each fully connected layer of a neural network stores a weight matrix W and a bias vector b. For an input x, the layer computes y = Wx + b. If the layer has 512 inputs and 256 outputs, W is 256 × 512—containing 131,072 learnable parameters. A deep network stacks dozens or hundreds of such matrices; large language models contain billions of parameters, almost all stored as matrices and tensors.

Batch Data as Matrices

Training data is processed in mini-batches for efficiency. Instead of one input vector at a time, a batch of B samples is stacked into a matrix X of shape B × features. One matrix multiplication WX^T (or equivalently batched operations) processes the entire batch in parallel on a GPU. This is why batch size directly affects memory usage: doubling the batch doubles the matrix height.

Example — MNIST Batch

The MNIST dataset uses 28 × 28 grayscale images. Flattened, each image is a 784-dimensional vector. A batch of 64 images forms a 64 × 784 matrix. Multiplying by a 128 × 784 weight matrix produces a 64 × 128 matrix of layer activations—64 outputs processed in one GPU kernel.

Attention Matrices in Transformers

Transformer architectures—the foundation of GPT, BERT, and similar models—rely heavily on attention. Queries, keys, and values are stored as matrices. Attention scores are computed as:

Attention(Q, K, V) = softmax(QK^T / √d_k) V

Here QK^T is a matrix multiplication producing an (sequence length × sequence length) matrix of compatibility scores. Each row answers: “How much should this token attend to every other token?” Without fluency in matrix dimensions, attention code is impossible to debug.

AI Concept	Matrix Role	Typical Shape
Neural network layer	Learned weight matrix transforms inputs to outputs	(outputs, inputs)
Mini-batch training	Rows = samples; columns = features	(batch_size, features)
Word embeddings	Lookup table: each row is one word’s vector	(vocab_size, embed_dim)
Self-attention scores	Pairwise token similarity after QK^T	(seq_len, seq_len)
Confusion matrix	Classification results: predicted vs. actual classes	(num_classes, num_classes)

Deep DiveMatrix multiplication—the operation that chains layers together—is covered in Matrix Multiplication. Higher-dimensional arrays generalize matrices in Tensor.

Comparing Matrix Operations

Addition

Requires identical dimensions
Element-wise: c_ij = a_ij + b_ij
Same shape as inputs
Used in residuals, gradient accumulation

Scalar Multiply

Any matrix shape
Element-wise: b_ij = c·a_ij
Same shape as input
Used in learning rate scaling

Transpose

Swaps rows and columns
m × n becomes n × m
Used in dot products, attention
(AB)^T = B^TA^T

Identity

Square matrix, ones on diagonal
AI = IA = A
Represents “no change” transform
Basis for inversion and eigenanalysis

Common Misconceptions

Misconception 1: “Matrix dimensions are columns × rows.”

Why people believe it: Some textbooks and image conventions list width before height.

Reality: Standard matrix notation is always rows × columns (m × n). NumPy and PyTorch follow this convention in shape.

Misconception 2: “You can always multiply any two matrices.”

Why people believe it: Framework code often broadcasts or batches operations automatically.

Reality: Standard matrix multiplication requires the inner dimensions to match. An (m × n) times (p × q) product exists only when n = p. Addition requires identical shapes.

Misconception 3: “Matrices are just spreadsheets with no geometric meaning.”

Why people believe it: Early exposure focuses on arithmetic drills.

Reality: Every matrix encodes a linear transformation. Understanding this viewpoint explains why neural network layers compose like functions and why ill-conditioned matrices cause training instability.

Quick Knowledge Check

Short Answer: What are the dimensions of a matrix with 4 rows and 7 columns? Answer: 4 × 7
True/False: Matrix addition requires both matrices to have the same dimensions. Answer: True
Multiple Choice: If A is 3 × 5, what is the shape of A^T? Answer: 5 × 3
Short Answer: What does the identity matrix do when multiplied with A? Answer: Returns A unchanged (IA = AI = A)
True/False: Scalar multiplication changes the shape of a matrix. Answer: False — only the entry values change
Multiple Choice: In a neural layer with 256 inputs and 128 outputs, what is the shape of the weight matrix? Answer: 128 × 256
Short Answer: In attention, what does the matrix QK^T represent? Answer: Pairwise compatibility scores between query and key tokens
True/False: A matrix is a linear transformation that maps vectors to vectors. Answer: True
Short Answer: Compute the sum: [[1,2],[3,4]] + [[5,6],[7,8]]. Answer: [[6,8],[10,12]]
Multiple Choice: A mini-batch of 32 samples with 784 features each is stored as a matrix of shape: Answer: 32 × 784

Key Takeaways

A matrix is a rectangular array of numbers; its dimension is rows × columns (m × n).
Matrices represent linear transformations that map n-dimensional inputs to m-dimensional outputs.
Addition and scalar multiplication operate element-wise; addition requires matching dimensions.
The transpose swaps rows and columns; if A is m × n, then A^T is n × m.
The identity matrix I leaves matrices and vectors unchanged under multiplication.
In AI, matrices store weights, batched data, embeddings, and attention scores—virtually every computation in deep learning involves them.
Shape errors are the most common beginner bug; always verify dimensions before multiplying.