← Master Index
Vol. 12 Module 12.2 Lecture

Dense Embeddings

Embeddings Deep Dive

How This Lesson Fits the Module & Volume

Module 12.1 locked down how text becomes IDs. Module 12.2 asks what those IDs (and whole passages) become for retrieval: continuous vectors. Building on Volume 11 embeddings and Sentence-BERT, dense embeddings are the default semantic search representation—compact float vectors ranked by cosine or dot product.

Later lectures contrast them with sparse signals and with bi- vs cross-encoders for ranking.

Learning Objectives

By the end of this lesson, students should be able to:

  • Define dense embeddings as fixed-length continuous vectors for text units.
  • Contrast token embeddings inside an LM with pooled sentence/document vectors.
  • Encode corpora with sentence-transformers and compute cosine similarity.
  • Explain why ANN indexes (HNSW, IVF) matter at retrieval scale.
  • List failure modes: paraphrases vs keywords, domain shift, length bias.
  • Position dense retrieval as the first stage of many RAG pipelines.
Definition

A dense embedding is a low-dimensional continuous vector (typically 256–1024 floats) produced by a neural encoder such that semantically related texts have high similarity (cosine / dot product). Unlike sparse bag-of-words vectors, most dimensions are nonzero and meaning is distributed across the vector.

Where Dense Vectors Come From

LevelExampleUse
TokenLM embedding table rowModel internals
Sentence / passageSBERT pooled outputSemantic search
MultimodalCLIP text/image towersCross-modal retrieval

Strengths

  • Paraphrase & synonym recall
  • Compact indexes
  • Multilingual models available

Weak Spots

  • Exact SKUs / IDs / rare terms
  • Opaque dimensions
  • Needs good training data

Ops Stack

  • Offline encode documents
  • ANN index (FAISS, etc.)
  • Online encode queries

Code: Dense Encode with sentence-transformers

from sentence_transformers import SentenceTransformer, util model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2") docs = [ "BPE merges frequent token pairs.", "Dense vectors enable semantic search.", "Pad tokens should be masked in loss.", ] q = "find meaning-based document retrieval" doc_emb = model.encode(docs, normalize_embeddings=True, convert_to_tensor=True) q_emb = model.encode(q, normalize_embeddings=True, convert_to_tensor=True) hits = util.semantic_search(q_emb, doc_emb, top_k=2)[0] for h in hits: print(h["score"], docs[h["corpus_id"]])

When Dense Wins

  • Natural language questions
  • Paraphrase-heavy corpora
  • Cross-lingual retrieval

When to Augment

  • Exact string / SKU match
  • Legal citation lookup
  • Hybrid with sparse (next)
Common Misconception

“Any Transformer hidden state is a great sentence embedding.” Untuned [CLS] or mean-pooled BERT often underperforms models trained with contrastive / siamese objectives. Use embedding-specialized checkpoints (or fine-tune) for retrieval.

Knowledge Check

  1. Short Answer: What similarity metrics are common for dense retrieval? Answer: Cosine similarity and/or dot product (often with normalized vectors).
  2. True/False: Dense embeddings are mostly zeros like one-hot BoW. Answer: False.
  3. Multiple Choice: SBERT-style models mainly produce: (a) parse trees, (b) fixed sentence vectors, (c) TF-IDF only. Answer: (b).
  4. Short Answer: Why encode documents offline? Answer: So queries only need one forward pass and can search a prebuilt ANN index.
  5. True/False: Dense retrieval always beats keyword search on SKU lookup. Answer: False.
  6. Multiple Choice: ANN indexes help when: (a) N is tiny, (b) corpus is large, (c) |V|=2. Answer: (b).
  7. Short Answer: Name a Python library for ready dense encoders. Answer: sentence-transformers.
  8. Short Answer: What is a typical failure of raw BERT CLS for search? Answer: Not trained for cosine ranking; weak semantic similarity.
  9. Multiple Choice: Dense dims are typically: (a) 2–8, (b) hundreds, (c) |V|. Answer: (b).
  10. True/False: Normalizing embeddings makes cosine equal to dot product. Answer: True (for L2-normalized vectors).

Key Takeaways

  • Dense embeddings map text to continuous vectors for semantic similarity.
  • Use embedding-trained encoders, not arbitrary LM states.
  • Scale with offline doc encoding + ANN search.
  • Weak on exact lexical IDs—plan hybrids.
  • Next: Sparse Embeddings.
Trainer’s Guide

Lab: Build a 50-doc FAQ index; compare MiniLM hits vs naive keyword overlap.

Prompt: When would 384-d be preferable to 1024-d embeddings in production?

Recap: Dense embeddings power semantic retrieval for RAG and search. Continue with Sparse Embeddings.