Vectors

How This Lesson Fits the Module

Volume 01 established what Machine Learning is—systems that learn patterns from data, optimize objectives, and generalize to unseen examples. See Machine Learning for that foundation. Volume 02 supplies the mathematics those systems run on.

Vectors are the first object in Module 2.1: Linear Algebra. Every feature row in a dataset, every word embedding, every gradient during training, and every pixel channel in an image is ultimately represented as a vector. Before matrices, dot products, and eigenvalues make sense, engineers must be fluent in what a vector is and how to manipulate it.

If Machine Learning is the paradigm, linear algebra is its native language—and vectors are its words.

Learning Objectives

By the end of this lesson, students should be able to:

Explain why vectors are the fundamental data structure underlying features, embeddings, and gradients in ML.
Define a vector as an ordered list of numbers with magnitude and direction.
Read and write standard vector notation, including column and row forms.
Perform vector addition and scalar multiplication correctly, with geometric interpretation.
Compute the L2 norm (Euclidean magnitude) and construct unit vectors.
Connect abstract vector operations to concrete AI examples: feature vectors, word embeddings, and image data.
Recognize how NumPy represents vectors in production ML code.
Identify common misconceptions about vector dimension, direction, and notation.

Introduction: The Language of Machine Learning

In the Machine Learning lecture, you learned that models consume features—input variables such as age, income, or word counts—and produce predictions. Behind every feature vector lies a mathematical object: a vector.

When a neural network processes an image, it does not see a picture. It sees a long ordered list of pixel intensities—a vector. When a language model represents the word “king,” it stores a list of 300 or 768 floating-point numbers—an embedding vector. When gradient descent updates model weights, each parameter change is a step along a direction in a high-dimensional vector space.

Vectors are not an abstract detour from AI engineering. They are the substrate on which modern ML is built.

Why Vectors Matter in Machine Learning

Three roles make vectors indispensable in production ML systems:

Role	What It Represents	ML Example
Features	Numeric encoding of an input instance	A house described as [sqft, bedrooms, year_built, latitude]
Embeddings	Dense learned representations of discrete or complex objects	Word2Vec vector for “algorithm” in 300 dimensions
Gradients	Direction and magnitude of parameter updates during optimization	∇_θL — the vector of partial derivatives of loss with respect to each weight

Engineering Principle

Every ML pipeline eventually reduces data to numbers arranged in order. Understanding vectors means understanding how your model “sees” the world—and how updates move through parameter space during training.

Defining a Vector

Definition — Vector

A vector is an ordered list of numbers that can represent both magnitude (length) and direction in space. Vectors live in ℝⁿ (read “R-n”)—the set of all n-tuples of real numbers, where n is the dimension.

Two equivalent views of the same object:

Algebraic — An ordered list: v = (3, −1, 4) or v = [3, −1, 4]^T
Geometric — An arrow from the origin to the point (3, −1, 4) in 3D space, with a definite length and pointing in a definite direction

In ML, the algebraic view dominates because dimensions routinely reach hundreds or thousands. The geometric view remains essential for intuition—especially in 2D and 3D—and generalizes cleanly to higher dimensions through the same operations.

Notation: Column vs Row Vectors

Precision in notation prevents silent bugs in matrix operations later in this module.

Column Vector (default in ML)

Written as a vertical stack. Standard in linear algebra and deep learning frameworks.

v = 3 −1 4

Dimension: 3 × 1. Lives in ℝ³.

Row Vector (transpose)

Written horizontally. The transpose of a column vector, denoted v^T.

v^T = [3, −1, 4]

Dimension: 1 × 3. Same numbers, different shape for multiplication.

Convention in this curriculum and in most ML literature: unless stated otherwise, v denotes a column vector. Row vectors appear explicitly as v^T. NumPy 1D arrays of shape (n,) behave like vectors but are neither strictly column nor row until reshaped—a detail that matters when multiplying matrices.

Coming UpMatrix-vector multiplication and the rules governing shapes are covered in Matrices and Matrix Multiplication.

Vector Addition

Vectors of the same dimension add component-wise:

a + b = a₁ + b₁ a₂ + b₂ ⋮ a_n + b_n

Example: [1, 2] + [3, 4] = [4, 6].

Geometric intuition: Place the tail of b at the head of a. The sum a + b is the arrow from the origin to the new head—the parallelogram rule. In ML, adding vectors combines information: averaging embeddings, accumulating gradient updates, or blending feature representations.

Scalar Multiplication

Multiplying a vector by a scalar (a single number) scales every component:

cv = (ca₁, ca₂, …, ca_n)

Example: 3 × [1, −2] = [3, −6].

Geometric intuition: If c > 0, the direction stays the same and the length stretches by factor c. If c < 0, the direction reverses. If c = 0, the result is the zero vector 0 = (0, 0, …, 0)—no direction, zero magnitude.

Learning rates in gradient descent are scalars that scale the gradient vector: θ_new = θ_old − η∇L. The scalar η controls step size; the vector ∇L controls direction.

Magnitude: The L2 Norm

Definition — Euclidean Norm (L2)

The magnitude or norm of vector v = (v₁, v₂, …, v_n) is:

‘v’ = √(v₁² + v₂² + … + v_n²) = √(v · v)

Also written ‘v’₂ or ‘v’_L2. It is the straight-line distance from the origin to the point v.

Example: For v = (3, −4), ‘v’ = √(9 + 16) = √25 = 5.

Other norms exist (L1, L∞) and appear in regularization. The L2 norm is the default for geometric distance and appears throughout loss functions, weight decay, and embedding normalization.

Unit Vectors

Definition — Unit Vector

A unit vector has magnitude exactly 1. Any nonzero vector v can be normalized:

û = v / ‘v’

The hat notation (û) often denotes a unit vector. Normalization preserves direction while setting length to 1.

Unit vectors matter in ML when comparing directions independent of magnitude—cosine similarity between embeddings uses normalized vectors. Gradient directions are often analyzed as unit vectors to separate which way parameters should move from how far.

Geometric Intuition in Practice

Even when working in 768 dimensions, geometric reasoning in 2D and 3D builds reliable intuition:

Same direction, different length — Scalar multiplication stretches or flips the arrow ↓ Tip-to-tail addition — Vector sum follows the parallelogram rule ↓ Norm = distance — Magnitude is how far the point is from the origin ↓ Unit vector = pure direction — Normalize to compare orientations only ↓ High dimensions — Same algebra; geometry is harder to draw but operations are identical

3Blue1Brown’s Essence of Linear Algebra series visualizes these ideas exceptionally well. Watch Chapter 1 (Vectors) and Chapter 2 (Linear Combinations) before or after this lecture to cement spatial intuition.

Vectors in AI: Three Concrete Examples

Example 1 — Feature Vectors (Tabular ML)

A credit-risk model might represent each applicant as:

x = [income, debt_ratio, credit_score, years_employed]^T ∈ ℝ⁴

Each component is a engineered feature. The model maps this 4-dimensional vector to a prediction (approve/deny or default probability). Thousands of applicants form a dataset of vectors—stored as rows in a matrix.

Example 2 — Word Embeddings (NLP)

Word2Vec, GloVe, and transformer models map each token to a dense vector. Similar words cluster in vector space:

“king” − “man” + “woman” ≈ “queen”

This famous analogy works because semantic relationships are encoded as vector offsets. Embeddings turn discrete symbols into continuous vectors that neural networks can process.

Example 3 — Images as Vectors (Computer Vision)

A 28×28 grayscale MNIST digit is reshaped into a 784-dimensional vector:

x ∈ ℝ⁷⁸⁴, where x_i ∈ [0, 1] is pixel intensity

Color images stack channels: a 224×224 RGB image becomes a vector of length 224 × 224 × 3 = 150,528. Convolutional networks learn hierarchical features, but at the input layer, the image is still a long ordered list of numbers.

NumPy: Vectors in Code

Python’s NumPy is the standard library for numerical vectors in ML. A vector is typically a 1D ndarray:

import numpy as np

v = np.array([3.0, -1.0, 4.0])       # shape (3,)
w = np.array([1.0, 2.0, 0.0])

# Vector addition and scalar multiplication
sum_vw = v + w                          # [4., 1., 4.]
scaled = 2.5 * v                        # [7.5, -2.5, 10.]

# L2 norm
magnitude = np.linalg.norm(v)           # 5.0

# Unit vector
unit_v = v / np.linalg.norm(v)          # [0.6, -0.2, 0.8]

# Column shape for matrix multiplication
v_col = v.reshape(-1, 1)                # shape (3, 1)

PyTorch and TensorFlow extend these ideas to GPU-accelerated tensors, but the underlying operations—addition, scaling, norms—remain vector operations. Fluency in NumPy transfers directly to deep learning frameworks.

Properties Engineers Should Know

Property	Statement	ML Relevance
Commutativity of addition	a + b = b + a	Order of combining gradient contributions does not matter
Distributivity	c(a + b) = ca + cb	Batch scaling distributes over per-sample gradients
Zero vector	v + 0 = v	Initialization often starts weights near the zero vector
Triangle inequality	‘a + b’ ≤ ‘a’ + ‘b’	Bounds on combined update magnitudes during optimization

Common Misconceptions

Misconception 1: “A vector is just a list—order doesn’t matter.”

Why people believe it: Programming lists and sets feel interchangeable in casual coding.

Reality: Vector components are ordered. Swapping positions changes the vector and, in ML, changes the feature meaning. [income, age] ≠ [age, income] unless your model was trained with that permutation.

Misconception 2: “You cannot visualize vectors beyond three dimensions, so geometry is useless in ML.”

Why people believe it: Textbooks draw arrows in 2D and 3D; embedding spaces have hundreds of dimensions.

Reality: The algebra generalizes perfectly. Norms, addition, and scalar multiplication work identically in ℝ⁷⁶⁸. Geometric intuition from low dimensions guides algorithm design even when visualization fails.

Misconception 3: “Row and column vectors are the same thing.”

Why people believe it: They contain the same numbers; NumPy 1D arrays blur the distinction.

Reality: Shape matters for matrix multiplication. A column vector (n × 1) times a row vector (1 × m) produces an (n × m) matrix. Confusing shapes causes dimension mismatch errors—among the most common bugs in ML code.

Misconception 4: “Bigger embedding vectors are always better.”

Why people believe it: State-of-the-art models use high-dimensional embeddings.

Reality: Higher dimension increases expressiveness but also memory, compute, and overfitting risk. Dimension is an engineering trade-off, not a quality score.

Quick Knowledge Check

Short Answer: What are two roles vectors play in Machine Learning? Answer: Any two from features, embeddings, gradients (or parameter updates).
True/False: Vector addition requires both vectors to have the same dimension. Answer: True.
Computation: Compute ‘v’ for v = (3, −4). Answer: 5.
Multiple Choice: Multiplying a vector by −2 changes: (a) only magnitude, (b) only direction, (c) both magnitude and direction, (d) neither. Answer: (c) — magnitude scales by 2 and direction reverses.
Short Answer: What is a unit vector? Answer: A vector with magnitude (L2 norm) equal to 1.
True/False: In standard ML notation, v without a transpose denotes a row vector. Answer: False — it denotes a column vector by convention.
Short Answer: A 28×28 grayscale image reshaped for a fully connected layer is a vector in what dimension? Answer: 784.
Multiple Choice: Which NumPy function computes the L2 norm? (a) np.sum, (b) np.linalg.norm, (c) np.dot, (d) np.abs. Answer: (b) np.linalg.norm.
Short Answer: What does scalar multiplication represent geometrically? Answer: Stretching or shrinking the vector’s length; negative scalars also reverse direction.
True/False: The zero vector has a well-defined direction. Answer: False — it has zero magnitude and no direction.

Key Takeaways

Vectors are ordered lists of numbers representing magnitude and direction—the atomic unit of data in ML.
Features, embeddings, and gradients are all vectors; fluency here is prerequisite for every later topic in this module.
Column vectors are the default notation; row vectors are transposes—shape matters for matrix operations.
Vector addition is component-wise; scalar multiplication scales all components and may reverse direction.
The L2 norm measures magnitude; unit vectors capture pure direction via normalization.
Geometric intuition from 2D/3D generalizes algebraically to the high-dimensional spaces where ML operates.
NumPy provides the practical toolkit: np.array, element-wise operations, and np.linalg.norm.
Volume 01 taught what ML does with data; this lecture begins the mathematics of how that data is represented.