Bi Encoder — Module 12.2

How This Lesson Fits the Module & Volume

Module 12.2 closes on the scalable twin of the cross-encoder: the bi-encoder (dual encoder). Query and document towers embed independently so document vectors can be indexed once and searched with ANN—the engine behind dense retrieval and SBERT-style systems.

Together with sparse/hybrid recall and cross-encoder precision, you now have the standard retrieve–fuse–rerank map. Volume 12 next shifts to inference optimization starting with KV Cache.

Learning Objectives

By the end of this lesson, students should be able to:

Define bi-encoders as dual towers producing comparable embeddings.
Explain asymmetric setups (query encoder ≠ doc encoder) when used.
Train/evaluate conceptually with contrastive / InfoNCE-style losses.
Encode and search with sentence-transformers bi-encoder models.
Compare latency/quality tradeoffs vs cross-encoders.
Design a full stack: bi-encoder retrieve → hybrid optional → cross rerank.

Definition

A bi-encoder (dual encoder) uses two encoder passes—often weight-tied towers—to map queries and documents into a shared vector space. Relevance is a cheap similarity (dot / cosine) between vectors, enabling offline document indexing and approximate nearest-neighbor retrieval.

Retrieve vs Rerank Roles

Stage	Model	Scale
Retrieve	Bi-encoder (+ sparse)	Millions of docs
Fuse	RRF / weights	Union of top lists
Rerank	Cross-encoder	Tens–hundreds

Bi-Encoder Pros

Cache all doc vectors
ANN in milliseconds
Batch encode corpora

Bi-Encoder Cons

No late interaction
Weaker than cross on hard pairs
Needs good negatives to train

Training Tips

In-batch negatives
Hard-negative mining
Domain fine-tuning

Code: Bi-Encoder Retrieval

from sentence_transformers import SentenceTransformer, util

# Bi-encoder: same model embeds queries and documents independently
bi = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
corpus = [
    "KV cache stores past keys and values for decoding.",
    "Cross-encoders jointly score query-document pairs.",
    "BPE merges frequent adjacent symbol pairs.",
]
corpus_emb = bi.encode(corpus, convert_to_tensor=True, normalize_embeddings=True)

query = "speed up autoregressive attention with cached states"
q_emb = bi.encode(query, convert_to_tensor=True, normalize_embeddings=True)
hits = util.semantic_search(q_emb, corpus_emb, top_k=2)[0]
for h in hits:
    print(float(h["score"]), corpus[h["corpus_id"]])

# Production: store corpus_emb in FAISS/HNSW; encode only the query online.

Production Fit

RAG first-stage recall
Semantic cache / dedup
Recommendation twins

Watch Outs

Stale indexes after doc edits
Query/doc distribution shift
Overlong docs need chunking

Common Misconception

“Bi-encoder and cross-encoder are competing products—pick one.” They solve different stages. Bi-encoders make search feasible; cross-encoders polish the shortlist. Most strong systems use both (often with sparse/hybrid recall as well).

Knowledge Check

Short Answer: Why can documents be embedded offline in a bi-encoder system? Answer: Their vectors do not depend on the query; only similarity is computed online.
True/False: Bi-encoders jointly attend across query and document tokens in one sequence. Answer: False—that is the cross-encoder.
Multiple Choice: Similarity is typically: (a) edit distance only, (b) cosine/dot on vectors, (c) JPEG size. Answer: (b).
Short Answer: Name a training ingredient that improves bi-encoders. Answer: Contrastive loss, in-batch negatives, and/or hard-negative mining.
True/False: ANN indexes pair naturally with bi-encoder document banks. Answer: True.
Multiple Choice: Cross-encoders usually sit: (a) before retrieving anything, (b) after a shortlist exists, (c) inside BPE merges. Answer: (b).
Short Answer: What is an asymmetric dual encoder? Answer: Different (or differently tuned) towers for queries vs documents.
Short Answer: Why chunk long documents? Answer: Embedding models have length limits; chunking preserves local topical match.
Multiple Choice: SBERT popularized bi-encoders for: (a) sentence similarity search, (b) JPEG compression, (c) k-means only. Answer: (a).
True/False: After Module 12.2, a common next systems topic is inference KV caching. Answer: True.

Key Takeaways

Bi-encoders embed queries and docs separately for scalable dense retrieval.
Cache document vectors; encode queries online; search with ANN.
Contrastive training and hard negatives drive quality.
Pair with hybrid recall and cross-encoder rerank for production stacks.
Next module: KV Cache in inference optimization.

Trainer’s Guide

Capstone: Teams diagram retrieve → fuse → rerank for a docs chatbot; assign models to each box.

Stretch: Fine-tune a MiniLM bi-encoder on a tiny domain pair dataset and measure recall@10 lift.

Recap: Bi-encoders make semantic search scalable; use them with hybrid and cross-encoder stages. Continue to KV Cache.