The Matrix Multiplication lecture showed that every entry in a product matrix is a dot product between a row of the left matrix and a column of the right matrix. That operation is the computational engine behind neural network layers, attention mechanisms, and embedding search—but matrix multiplication is only useful once you understand what a single dot product means.
This lecture defines the dot product algebraically and geometrically, connects it to angles and projections, and shows why it is the default similarity measure in modern AI—from word embeddings to the attention scores inside Transformers. The next lecture, Cross Product, covers a different vector product that applies only in three dimensions and produces a perpendicular vector rather than a scalar.
Learning Objectives
By the end of this lesson, students should be able to:
- Compute the dot product of two vectors using the component-wise sum definition.
- Interpret the dot product geometrically as projection length scaled by vector magnitude.
- Relate the dot product to the angle between vectors via a · b = ‖a‖ ‖b‖ cos θ.
- Explain why embedding similarity in NLP and retrieval systems is often measured with dot products or cosine similarity.
- Derive cosine similarity from the dot product and state when each measure is preferred.
- Recognize dot products inside matrix multiplication and as the raw scores in scaled dot-product attention.
- Distinguish the dot product (scalar output) from the cross product (vector output, 3D only).
Introduction: One Operation, Many Roles
Linear algebra gives us several ways to combine vectors. Addition places vectors tip-to-tail. Scalar multiplication stretches or flips a vector. The dot product—also called the inner product or scalar product—is different: it takes two vectors and returns a single number.
That number is deceptively simple. It tells you how much two vectors point in the same direction. It measures overlap, alignment, and—after normalization—semantic similarity. Every time a language model decides which words to attend to, a recommendation engine ranks nearest neighbors, or a neural network layer transforms an input, dot products are being evaluated millions of times per second.
Mastering the dot product is not optional for AI engineers. It is the bridge between the geometry of vectors and the arithmetic of matrices.
Algebraic Definition
Given two vectors a = (a1, a2, …, an) and b = (b1, b2, …, bn) in ℝn, the dot product is:
a · b = a1b1 + a2b2 + … + anbn = ∑i=1n aibi
Both vectors must have the same dimension. The result is a scalar, not a vector.
When vectors are written as columns, the dot product is equivalent to matrix multiplication with a transpose:
a · b = aTb
This notation appears constantly in machine learning code, where a batch of dot products becomes a single matrix multiply.
Let a = (3, −1, 2) and b = (1, 4, −2).
a · b = (3)(1) + (−1)(4) + (2)(−2)
= 3 − 4 − 4
= −5
The negative result means the vectors point more toward opposite half-spaces than the same half-space—we will make this precise with angles below.
Basic Properties
The dot product behaves predictably under standard vector operations. These properties are used in almost every proof and implementation:
| Property | Formula | Meaning |
|---|---|---|
| Commutative | a · b = b · a | Order does not matter |
| Distributive | a · (b + c) = a · b + a · c | Dot product distributes over addition |
| Scalar pull-out | (ka) · b = k(a · b) | Scaling one vector scales the result |
| Self-dot (squared norm) | a · a = ‖a‖2 ≥ 0 | Equals squared length; zero only for the zero vector |
| Orthogonality test | a · b = 0 ⇔ a ⊥ b (non-zero vectors) | Zero dot product means perpendicular |
Geometric Meaning: Projection and Angle
The algebraic definition sums products of components. The geometric definition reveals why that sum measures alignment.
For non-zero vectors a and b with angle θ between them (0° ≤ θ ≤ 180°):
a · b = ‖a‖ ‖b‖ cos θ
Equivalently: cos θ = (a · b) / (‖a‖ ‖b‖)
Read this formula in three cases that engineers use daily:
- Same direction (θ = 0°): cos 0° = 1, so a · b = ‖a‖ ‖b‖ (maximum positive value for given lengths).
- Perpendicular (θ = 90°): cos 90° = 0, so a · b = 0 (no alignment).
- Opposite direction (θ = 180°): cos 180° = −1, so a · b = −‖a‖ ‖b‖ (maximum negative value).
Projection onto a Vector
Place the tail of b at the origin. Drop a perpendicular from the tip of a onto the line through b. The scalar projection of a onto b is the signed length of that shadow along b’s direction:
compb(a) = (a · b) / ‖b‖
The vector projection is that scalar times the unit direction of b:
projb(a) = ((a · b) / ‖b‖2) b
Find the angle between a = (1, 2) and b = (2, 1).
a · b = (1)(2) + (2)(1) = 4
‖a‖ = √(12 + 22) = √5
‖b‖ = √(22 + 12) = √5
cos θ = 4 / (√5 · √5) = 4/5 = 0.8
θ = arccos(0.8) ≈ 36.9°
Connection to Matrix Multiplication
Matrix multiplication is a grid of dot products. If C = AB where A is m × n and B is n × p, then each entry is:
Cij = (row i of A) · (column j of B)
A fully connected neural network layer computes y = Wx + b. Each output yi is the dot product of row i of the weight matrix W with the input vector x, plus a bias. Understanding one dot product explains one neuron; understanding matrix multiplication explains the entire layer.
Similarity in Embeddings
In modern AI, words, sentences, images, and users are represented as embedding vectors—high-dimensional points learned so that related items land near each other. “Near” is measured by dot product or a close relative.
Consider three simplified 3D word embeddings:
| Word | Embedding vector |
|---|---|
| king | (0.9, 0.3, 0.1) |
| queen | (0.8, 0.4, 0.2) |
| apple | (0.1, 0.1, 0.9) |
Computing dot products (exact values depend on training; this illustrates the pattern):
- king · queen is relatively large—both vectors share similar components in the first dimensions (royalty-related features).
- king · apple is relatively small—they align weakly; the third component dominates for apple but not for king.
Retrieval systems (search, RAG, recommendation) often rank candidates by dot product between a query embedding and document embeddings. Higher dot product → higher relevance score → surfaced first.
Dot product rewards both alignment in direction and large magnitudes. If embeddings are trained so that relevance corresponds to pointing the same way with confident (large) activations, dot product is a natural match. It is also extremely fast: GPUs are optimized for matrix multiplication, which is batched dot products.
Cosine Similarity: Normalizing the Dot Product
Sometimes magnitude should not affect similarity. A short document and a long document about the same topic may have embeddings of different lengths. Cosine similarity removes length from the comparison and keeps only the angle.
cos_sim(a, b) = (a · b) / (‖a‖ ‖b‖) = cos θ
Values range from −1 (opposite) through 0 (orthogonal) to +1 (identical direction). For many NLP embeddings, values fall between 0 and 1 because components are non-negative after ReLU or similar activations.
Dot Product
- Depends on vector magnitudes
- Favors longer, confident vectors
- Default in many retrieval indexes (FAISS inner product)
- Used when scale carries meaning
Cosine Similarity
- Magnitude-invariant (only angle matters)
- Fair comparison across short and long texts
- Common in semantic search APIs
- Used when direction is the signal
Let a = (1, 1) and b = (2, 2). Note b = 2a—same direction, different length.
a · b = (1)(2) + (1)(2) = 4
cos_sim(a, b) = 4 / (√2 · √8) = 4 / 4 = 1
Dot product is 4 (large, because b is long). Cosine similarity is 1 (perfect alignment regardless of length). If you only care about “same topic,” cosine similarity is often the better choice.
If all embeddings are L2-normalized to unit length (‖v‖ = 1), dot product and cosine similarity become identical. Many production pipelines normalize once at index time so a single fast inner-product search implements cosine similarity.
Preview: Dot Products in Attention Scores
Transformers—the architecture behind GPT, BERT, and most modern language models—use scaled dot-product attention. The idea previewed here; full derivations appear in later volumes on deep learning.
Each token is represented by three learned vectors: a query (q), a key (k), and a value (v). To decide how much token i should attend to token j, the model computes a dot product between the query of i and the key of j:
score(i, j) = qi · kj
High score → strong match between “what I am looking for” (query) and “what I offer” (key). Scores are scaled by √d (dimension of keys) to prevent extreme values, then passed through softmax to produce weights that sum to 1. The output is a weighted sum of value vectors.
Attention(Q, K, V) = softmax(QKT / √dk) V
The matrix QKT is a table of dot products: row i, column j is qi · kj. This is exactly the matrix-multiplication pattern from the previous lecture, applied to learned semantic vectors instead of raw coordinates.
A concrete intuition: when the model processes the sentence “The cat sat on the mat,” the query for sat may dot-product strongly with the key for cat (subject–verb link) and weakly with mat. Softmax converts those dot products into attention weights that blend value vectors, letting sat “look at” relevant context.
Dot Product vs Cross Product
Students often confuse the two vector products. They serve different purposes:
Dot Product
- Input: two vectors in ℝn
- Output: a scalar
- Measures alignment / projection
- Core to AI: embeddings, layers, attention
Cross Product
- Input: two vectors in ℝ3 only
- Output: a vector perpendicular to both
- Measures oriented area / rotation
- Common in 3D graphics and physics
Common Misconceptions
Reality: The dot product always returns a scalar. The operation that returns a perpendicular vector in 3D is the cross product.
Reality: Negative means they point toward opposite half-spaces—they are strongly related, but inversely. Unrelated (orthogonal) vectors have dot product zero.
Reality: They rank identically only when vector norms are fixed or comparable. With varying magnitudes, dot product favors longer vectors; cosine similarity does not.
Reality: At its core, attention is dot products between queries and keys, normalized into weights. The surrounding machinery (multi-head, feed-forward layers) adds capacity, but the scoring mechanism starts here.
Quick Knowledge Check
- Short Answer: Compute (2, −1, 3) · (1, 4, 1). Answer: 2 − 4 + 3 = 1
- True/False: The dot product is commutative. Answer: True — a · b = b · a
- Short Answer: If a · b = 0 for non-zero vectors, what is the angle between them? Answer: 90° (orthogonal)
- Multiple Choice: In C = AB, the entry Cij equals: Answer: dot product of row i of A and column j of B
- Short Answer: Write cosine similarity in terms of dot product. Answer: (a · b) / (‖a‖ ‖b‖)
- True/False: Dot product and cross product both return scalars. Answer: False — cross product returns a vector
- Short Answer: If a = (3, 0) and b = (4, 0), what is a · b? Answer: 12
- Multiple Choice: In attention, score(i, j) is computed as: Answer: dot product of query i and key j
- Short Answer: When are dot product and cosine similarity equivalent for ranking? Answer: When all vectors have the same magnitude (e.g., L2-normalized to unit length)
- True/False: A larger dot product always means smaller angle. Answer: False for negative dot products — angle > 90° gives negative values; among positive dot products with fixed lengths, larger means smaller angle
Key Takeaways
- The dot product sums component-wise products and returns a scalar measuring how much two vectors align.
- Geometrically, a · b = ‖a‖ ‖b‖ cos θ, linking algebra to angle and projection.
- Matrix multiplication entries are dot products; neural network layers are batched dot products plus bias.
- Embedding similarity in search and NLP uses dot products or cosine similarity—the latter normalizes away vector length.
- When embeddings are unit-normalized, dot product equals cosine similarity.
- Transformer attention scores begin as scaled dot products between query and key vectors.
- The cross product (next lecture) is a different operation: 3D only, vector output, perpendicular to both inputs.
Further Reading & References
Textbooks
- Linear Algebra and Its Applications — Gilbert Strang. Clear geometric treatment of inner products and projections.
- Deep Learning — Goodfellow, Bengio, Courville. Chapter 2 covers vectors, dot products, and matrix operations for ML.
Research & Architecture
- Attention Is All You Need — Vaswani et al. (2017). Introduces scaled dot-product attention.
- Word2Vec — Mikolov et al. (2013). Embedding vectors whose dot products encode semantic relationships.
Tools
- NumPy
np.dot/@operator — Standard dot product and matrix multiply in Python - FAISS (Meta) — Billion-scale similarity search using inner products on GPUs
Teaching strategy: Draw two vectors on a whiteboard. Show the projection shadow first, then reveal that its length times ‖b‖ equals the dot product. Students remember the geometry.
Hands-on idea: In Python, embed three sentences with any small model; compute pairwise dot products and cosine similarities. Ask students to predict rankings before running code.
Discussion prompt: A search engine returns different top results using dot product vs cosine similarity on the same embeddings. When would that happen?
Bridge to next lecture: Emphasize that cross product is for 3D geometry and graphics; dot product is the one that powers AI pipelines students will build daily.