Matrix Multiplication

How This Lesson Fits the Module

Earlier lectures introduced vectors, matrices, scalars, and tensors as objects. Matrix multiplication is the operation that makes those objects work together—and the operation that neural networks execute billions of times per second.

Every fully connected layer computes W·x + b. Every attention block, convolution (when unfolded), and transformer block ultimately reduces to matrix multiplies batched across samples. If you understand the dimension rules and algebraic properties taught here, backpropagation, GPU kernels, and framework error messages become legible instead of mysterious.

Learning Objectives

By the end of this lesson, students should be able to:

State the compatibility rule for matrix multiplication: inner dimensions must match; outer dimensions give the result shape.
Compute products of small matrices by hand using the row-by-column (dot-product) rule.
Explain why matrix multiplication is associative but generally not commutative, with concrete counterexamples.
Interpret a matrix–vector product as a linear transformation and express a neural network layer as W·x + b.
Describe how composing transformations corresponds to multiplying matrices in a specific order.
Extend single-vector multiplication to batch matrix multiplication as used in deep learning frameworks.
Diagnose common shape-mismatch errors when multiplying matrices in code.

Introduction: The Operation That Powers Deep Learning

Adding matrices is element-wise: same shape, add corresponding entries. Multiplying matrices is fundamentally different. It is not element-wise. It encodes how linear transformations compose—how one change of coordinates follows another, how a layer maps input features to output activations, how a rotation followed by a scaling is captured in a single matrix product.

Scalar multiplication stretches a vector uniformly. Matrix multiplication can stretch, rotate, shear, project, and mix dimensions in ways scalars cannot. That expressive power is exactly why weight matrices sit at the center of neural network design.

This lecture builds the rules, works through small numeric examples, and connects the algebra directly to the code you will write in PyTorch, TensorFlow, JAX, and NumPy.

Definition: Row Meets Column

Definition — Matrix Multiplication

Let A be an m × n matrix and B be an n × p matrix. The product C = AB is an m × p matrix whose entry in row i, column j is:

c_ij = (row i of A) · (column j of B)

That dot product sums the pairwise products of corresponding entries: c_ij = Σ_k a_ik b_kj.

Read the definition twice. Each entry of the result is a dot product between one row of the left matrix and one column of the right matrix. The next lecture on the dot product unpacks that inner operation in detail; here we treat it as the building block of matrix multiply.

The Compatibility Rule: Rows × Columns

Before any arithmetic, check shapes. Matrix multiplication is only defined when the number of columns of the left factor equals the number of rows of the right factor.

Left Factor	Right Factor	Product Shape	Valid?
(m × n)	(n × p)	(m × p)	Yes
(2 × 3)	(3 × 4)	(2 × 4)	Yes
(2 × 3)	(2 × 3)	—	No — inner 3 ≠ 2
(4 × 1)	(1 × 3)	(4 × 3)	Yes — outer product pattern

Memory Aid Write shapes as (rows × cols). The two inner numbers must match; the two outer numbers become the result shape: (m × n)(n × p) → (m × p).

Common Error

Students often try to multiply matrices of identical shape, assuming element-wise rules carry over. A (2 × 3) matrix cannot multiply another (2 × 3) matrix. The inner dimensions 3 and 2 do not match. Frameworks will raise a shape error—or silently broadcast in unrelated operations if you confuse * (element-wise) with @ (matrix multiply).

Worked Example: A 2 × 2 Product

Let

A = [ 1  2 ]       B = [ 5  6 ]
    [ 3  4 ]           [ 7  8 ]

Both are 2 × 2, so AB is 2 × 2. Compute entry by entry:

c₁₁ = (row 1 of A) · (col 1 of B) = 1·5 + 2·7 = 19
c₁₂ = 1·6 + 2·8 = 22
c₂₁ = 3·5 + 4·7 = 43
c₂₂ = 3·6 + 4·8 = 50

AB = [ 19  22 ]
     [ 43  50 ]

Example — Non-Square Matrices

Let A be 2 × 3 and B be 3 × 2. Inner dimension 3 matches, so AB is 2 × 2.

A = [ 1  0  2 ]       B = [ 1  2 ]
    [ 3  1  0 ]           [ 0  1 ]
                          [ 4  0 ]

c₁₁ = 1·1 + 0·0 + 2·4 = 9 | c₁₂ = 1·2 + 0·1 + 2·0 = 2

c₂₁ = 3·1 + 1·0 + 0·4 = 3 | c₂₂ = 3·2 + 1·1 + 0·0 = 7

AB = [ 9  2 ]
     [ 3  7 ]

Note: BA would be (3 × 2)(2 × 3) = (3 × 3)—a different shape entirely. Order matters.

Associative, Not Commutative

Matrix multiplication behaves differently from ordinary scalar multiplication in two critical ways. Engineers who internalize these properties avoid subtle bugs in transformation pipelines and neural network wiring.

Associative (order of grouping)

(AB)C = A(BC)

When three matrices are compatible, you may parenthesize freely. This is why deep frameworks can fuse chains of matmuls and why GPU libraries optimize A(B(Cx)) as a single pipeline.

Not Commutative (order of factors)

AB ≠ BA (in general)

Swapping factors usually changes the result—or makes the product undefined. “Rotate then scale” is not the same as “scale then rotate.”

Verifying Associativity

A = [ 1  0 ]    B = [ 0  1 ]    C = [ 1  1 ]
    [ 0  2 ]        [ 1  0 ]        [ 0  1 ]

First compute AB = [[0, 1], [2, 0]], then (AB)C = [[0, 1], [2, 2]].

Alternatively, compute BC = [[0, 1], [1, 1]], then A(BC) = [[0, 1], [2, 2]]. Same result. Parentheses moved; answer unchanged.

A Commutativity Counterexample

A = [ 1  2 ]    B = [ 0  1 ]
    [ 0  1 ]        [ 1  0 ]

AB = [[2, 1], [1, 0]] but BA = [[0, 1], [1, 2]].

Same matrices, reversed order—completely different outputs. Never assume you can swap factors to “simplify” an expression.

Identity Matrix

The identity matrix I (ones on the diagonal, zeros elsewhere) satisfies AI = IA = A. It plays the role of “1” for matrix multiplication—but it does not make multiplication commutative.

Matrix–Vector Multiplication: One Layer, One Input

A vector is a matrix with one column (an n × 1 matrix). Multiplying an m × n matrix by an n × 1 vector yields an m × 1 vector. Each output component is a dot product between one row of the matrix and the input vector.

Geometrically, a matrix acts as a linear transformation: it maps an input vector in ℝⁿ to an output vector in ℝ^m. Rotation, reflection, scaling along axes, and projection are all representable this way.

The Neural Network Layer: W·x + b

A fully connected (dense) layer is matrix multiplication plus a bias vector:

y = Wx + b

x — input vector, shape (n,) or (n × 1): one sample’s features
W — weight matrix, shape (m × n): m neurons, each with n weights
b — bias vector, shape (m,): one offset per neuron
y — pre-activation output, shape (m,): weighted sums before ReLU, sigmoid, etc.

Example — One Forward Pass

Suppose a layer has 2 inputs and 3 neurons:

W = [  0.5  -0.2 ]       x = [ 1.0 ]       b = [  0.1 ]
    [  0.1   0.8 ]           [ 2.0 ]           [ -0.1 ]
    [ -0.3   0.4 ]                               [  0.2 ]

Wx (row-by-column dot products):

Neuron 1: 0.5·1.0 + (−0.2)·2.0 = 0.1
Neuron 2: 0.1·1.0 + 0.8·2.0 = 1.7
Neuron 3: (−0.3)·1.0 + 0.4·2.0 = 0.5

Add bias: y = Wx + b = [0.1 + 0.1, 1.7 − 0.1, 0.5 + 0.2] = [0.2, 1.6, 0.7].

Each row of W is one neuron’s weight vector. Matrix multiply computes all neurons in parallel—exactly what GPUs exploit.

Framework Convention

PyTorch stores nn.Linear weights as shape (out_features, in_features) and computes y = x @ W.T + b. NumPy uses W @ x + b when W is (m, n) and x is (n,). The math is identical; only storage layout differs. Always check documentation before transposing.

Composition of Transformations

Applying transformation B to vector x, then transformation A to the result, yields:

A(Bx) = (AB)x

The composite transformation is a single matrix AB—but notice the order: B is applied first (closest to x), then A. Matrix multiplication reads right-to-left on the vector.

Example — Scale Then Rotate vs Rotate Then Scale

S = [ 2  0 ]    (scale x by 2, y by 0.5)       R = [ 0  -1 ]
    [ 0  0.5 ]                                       [ 1   0 ]    (90° CCW rotation)

x = [ 1 ]    (point on the x-axis)
    [ 0 ]

Scale then rotate: R(Sx) = R([2, 0]) = [0, 2]. Composite matrix RS.

Rotate then scale: S(Rx) = S([0, 1]) = [0, 0.5]. Composite matrix SR.

Different paths, different endpoints. In a neural network, layer order is not interchangeable—reordering layers changes the function unless the matrices satisfy special commutation conditions (rare).

Deep networks stack many layers: y = W_L(…W₂(W₁x + b₁) + b₂…) + b_L. Linear sub-parts compose as matrix products; nonlinear activations break pure associativity between layers but the linear blocks still obey the algebra taught here.

Batch Matrix Multiplication

Training never uses one sample at a time. A mini-batch stacks B input vectors as rows of a matrix X with shape (B × n). One matrix multiply applies the same weights to every sample simultaneously.

Y = XW + b (broadcast bias across rows)

X — batch of inputs, shape (B × n)
W — same weight matrix as before, shape (m × n) or transposed per framework
Y — batch of outputs, shape (B × m)

Each row of Y is the layer output for one sample. The operation is identical to looping over samples—but implemented as a single batched GEMM (General Matrix Multiply) on the GPU, often 10–100× faster.

Example — Batch of Three Samples

X = [ 1  2 ]       W = [  1   0 ]       b = [ 0  0 ]
    [ 3  4 ]           [ -1   1 ]
    [ 5  6 ]

Row 1: [1, 2] @ W = [1·1 + 2·(−1), 1·0 + 2·1] = [−1, 2]

Row 2: [3, 4] @ W = [3·1 + 4·(−1), 3·0 + 4·1] = [−1, 4]

Row 3: [5, 6] @ W = [5·1 + 6·(−1), 5·0 + 6·1] = [−1, 6]

Y = XW + b = [ -1   2 ]
             [ -1   4 ]
             [ -1   6 ]

Three forward passes, one matrix operation. This is the computational pattern behind torch.nn.Linear, tf.keras.layers.Dense, and every transformer MLP block at batch scale.

Higher-Dimensional Batches (Tensors)

When inputs carry extra dimensions—sequence length, image height and width, attention heads—frameworks use batched matrix multiplication on the trailing two dimensions. A tensor of shape (B, T, n) multiplied by weights (m, n) yields (B, T, m): the same matmul applied independently at each batch index and time step. The tensor lecture’s shape intuition pays off directly here.

Single sample — y = Wx + b shapes: (m×n)(n×1) + (m) ↓ Mini-batch — Y = XW + b shapes: (B×n)(n×m) → (B×m) ↓ Tensor batch — matmul on last two dims e.g. (B,T,n) → (B,T,m)

Computational Cost

Multiplying an m × n matrix by an n × p matrix requires on the order of mnp multiply-add operations. For a layer with n inputs and m neurons processing B samples, cost scales as Bmn. This is why widening layers (larger m or n) and increasing batch size dominate GPU memory and FLOP budgets.

Modern hardware is built for this pattern. CUDA cores, tensor cores, and TPUs are, at their core, matrix-multiplication engines. Understanding matmul is understanding what neural network training spends its time doing.

Common Misconceptions

Misconception 1: “Matrix multiplication is element-wise, like addition.”

Why people believe it: Both operations use the × symbol in casual notation.

Reality: Element-wise multiply is the Hadamard product (A * B in NumPy). Matrix multiply (A @ B) uses row-column dot products and strict shape rules.

Misconception 2: “AB = BA always.”

Why people believe it: Scalar multiplication commutes, and the notation looks symmetric.

Reality: Order encodes composition order. Reversing factors usually changes—or undefined—the result.

Misconception 3: “If AB is defined, then BA is too.”

Why people believe it: Students check one order, assume symmetry of validity.

Reality: (2 × 3)(3 × 2) is valid; (3 × 2)(2 × 3) is also valid but yields a different shape. (2 × 3)(2 × 3) is simply undefined.

Misconception 4: “Batching changes the math.”

Why people believe it: Higher-dimensional tensors look intimidating.

Reality: Batching stacks independent samples. Each row (or trailing slice) obeys the same W·x + b rule. Frameworks parallelize; the algebra does not change.

Quick Knowledge Check

Short Answer: What must match for AB to be defined? Answer: Columns of A must equal rows of B (inner dimensions).
Compute: If A is 3 × 4 and B is 4 × 2, what is the shape of AB? Answer: 3 × 2
True/False: Matrix multiplication is commutative. Answer: False
True/False: (AB)C = A(BC) when products are defined. Answer: True (associative)
Short Answer: In y = Wx + b, what does each row of W represent? Answer: One neuron’s weights over all inputs
Compute: [1, 2] · [3, 4]^T as a row-vector times column-vector (1 × 2)(2 × 1). Answer: 1·3 + 2·4 = 11
Short Answer: If you apply transformation B then A to x, what matrix represents the composite? Answer: AB, since A(Bx) = (AB)x
Multiple Choice: X has shape (32, 128), W has shape (64, 128). What is the shape of X @ W^T? Answer: (32, 64)
True/False: Element-wise multiplication and matrix multiplication are the same operation. Answer: False
Short Answer: Why do GPUs accelerate neural network training? Answer: They parallelize massive numbers of matrix multiply-add operations (batched GEMM)

Key Takeaways

Matrix multiply computes dot products of rows (left) with columns (right): c_ij = Σ_k a_ik b_kj.
Compatibility rule: (m × n)(n × p) → (m × p). Inner dimensions must match.
Associative: group freely. Not commutative: order of factors matters.
A neural network layer is y = Wx + b—one matrix multiply plus bias.
Composing linear maps multiplies matrices: applying B then A gives AB (read right-to-left on the vector).
Batching stacks samples as rows of X; Y = XW + b processes the entire mini-batch in one GEMM.
Frameworks may transpose stored weights, but the underlying math is always row-column dot products.