Indexing — Module 14.1

How This Lesson Fits the Module & Volume

Your knowledge base chunks and embeddings still need a data structure for fast vector search. Indexing here means building ANN (and sparse) structures—HNSW, IVF, PQ, inverted lists—not the module overview page. This closes Module 14.1 and hands off to Module 14.2 FAISS for hands-on vector indexes.

Learning Objectives

By the end of this lesson, students should be able to:

Define indexing as building search structures over embeddings (and terms).
Contrast exact search with ANN families (HNSW, IVF, PQ).
State the recall–latency–memory tradeoff when choosing index params.
Describe rebuild vs incremental upsert workflows after KB changes.
List what to version with an index (model id, metric, params, digests).
Preview why FAISS (and managed vector DBs) implement these ideas.

Definition

Indexing (for RAG retrieval) is the process of organizing vectors and/or sparse terms into structures that support efficient similarity or keyword queries—often approximate nearest neighbor (ANN) indexes plus optional inverted indexes for hybrid search.

ANN Families (Conceptual)

Family	Idea	Tune for
Flat / exact	Score all vectors	Gold recall; small N
HNSW	Graph walk neighbors	High recall, RAM-heavy
IVF	Cluster + search nprobe lists	Large N, tunable probe
PQ / compression	Codebooks shrink vectors	Memory at some recall cost

Build time

Train clusters / graphs
Can be hours at scale
Version artifacts

Query time

efSearch / nprobe
Latency vs recall
Monitor p95

Updates

Upsert if supported
Else periodic rebuild
Delete stale IDs

Code: Index Build Checklist (Conceptual)

import json
import numpy as np
from datetime import datetime, timezone

def build_index_manifest(vectors: np.ndarray, model_id: str, metric: str, params: dict) -> dict:
    """Persist this beside the ANN files so query nodes stay compatible."""
    assert vectors.ndim == 2
    return {
        "built_at": datetime.now(timezone.utc).isoformat(),
        "model_id": model_id,
        "metric": metric,          # "cosine" | "ip" | "l2"
        "dim": int(vectors.shape[1]),
        "nrows": int(vectors.shape[0]),
        "params": params,          # e.g. {"type": "hnsw", "M": 32, "efConstruction": 200}
        "normalized": True,
    }

X = np.random.default_rng(0).normal(size=(1000, 384)).astype("float32")
X /= np.linalg.norm(X, axis=1, keepdims=True) + 1e-12
manifest = build_index_manifest(
    X, "sentence-transformers/all-MiniLM-L6-v2", "cosine",
    {"type": "hnsw", "M": 32, "efConstruction": 200},
)
print(json.dumps(manifest, indent=2))
# Next module: construct the real FAISS/HNSW structure from X + manifest.

Strengths of ANN indexing

Sub-linear query time at scale
Tunable recall/latency
Enables production RAG SLOs

Tradeoffs

Approximate misses possible
Build complexity & RAM
Param tuning is corpus-specific

Common Misconception

“Once the index is built, we never touch it.” KB updates, embedding-model upgrades, and deletes all invalidate or stale-out parts of the index. Treat indexing as a recurring job with manifests, smoke-test recall, and rollback.

Knowledge Check

Short Answer: What does indexing mean in this lecture? Answer: Building ANN/sparse search structures over embeddings/terms for fast retrieval.
True/False: ANN always returns exact nearest neighbors. Answer: False.
Multiple Choice: HNSW is: (a) a graph-based ANN index, (b) a CSS unit, (c) a tokenizer. Answer: (a).
Short Answer: Name one IVF knob. Answer: nprobe (or number of lists/clusters).
True/False: Index manifests should record embedding model id and metric. Answer: True.
Multiple Choice: PQ primarily helps: (a) memory compression, (b) font kerning, (c) DNS. Answer: (a).
Short Answer: Why rebuild after model change? Answer: Vectors live in a new space; old index is incompatible.
True/False: Flat exact search is useless for evaluation. Answer: False—it is a recall gold standard for small/medium N.
Multiple Choice: Next module starts with: (a) FAISS, (b) Vol. 1 only, (c) printers. Answer: (a).
Short Answer: What tradeoff do ANN params control? Answer: Recall vs latency (and often memory).

Key Takeaways

Indexing builds the ANN/sparse structures that make RAG searchable at scale.
Tune recall–latency–memory; version manifests with model + metric.
Treat rebuilds and deletes as first-class ops.
Next module: FAISS.

Trainer’s Guide

Lab: On a 10k toy matrix, time brute-force top-10 vs a mocked ANN that probes 5% of rows; discuss missed neighbors.

Prompt: Write the rollback plan if a new HNSW build drops recall@10 by 8%.

Recap: Indexing closes Module 14.1’s path from RAG concepts to scalable search. Continue with FAISS in Module 14.2.