← Master Index
Vol. 12 Module 12.2 Lecture

Cross Encoder

Embeddings Deep Dive

How This Lesson Fits the Module & Volume

After hybrid candidate generation, quality often hinges on reranking. A cross-encoder jointly consumes the query and a document in one Transformer forward pass, attending across both—more accurate than cosine over independent vectors, but too slow to score millions of docs online.

Volume 11’s Sentence-BERT contrast previewed this tradeoff; here we operationalize cross-encoders as the precision stage before the next lecture’s scalable bi-encoder.

Learning Objectives

By the end of this lesson, students should be able to:

  • Describe cross-encoder input as a packed query–document pair.
  • Explain why cross-attention yields stronger relevance scores than bi-encoder cosine.
  • Use CrossEncoder from sentence-transformers to rerank a shortlist.
  • Place cross-encoders after retrieval (top-50/100), not as sole search.
  • Estimate latency: O(k) forward passes per query for k candidates.
  • Choose pointwise vs pairwise training objectives at a conceptual level.
Definition

A cross-encoder is a relevance model that encodes the query and a candidate document together (typically as a single sequence with a separator), producing a scalar score from joint self-attention. It does not produce cacheable independent document embeddings for ANN search.

Architecture Contrast

Cross-encoderBi-encoder
Input[query; doc] jointlyquery & doc separately
OutputRelevance scoreTwo vectors
Doc cache?No (per pair)Yes
Best roleRerank top-kRetrieve from corpus

Accuracy

  • Full cross-attention
  • Nuanced mismatches caught
  • SOTA rerank quality

Cost

  • One pass per pair
  • No ANN over corpus
  • Batch pairs on GPU

Pipeline

  • Retrieve k=50–200
  • Cross-encode scores
  • Send top-n to LLM

Code: Rerank with CrossEncoder

from sentence_transformers import CrossEncoder model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2") query = "How does byte-level BPE avoid UNKs?" candidates = [ "Byte-level BPE starts from all 256 bytes.", "Dropout randomly zeros activations.", "WordPiece uses ## continuation markers.", ] pairs = [[query, c] for c in candidates] scores = model.predict(pairs) ranked = sorted(zip(scores, candidates), reverse=True) for s, c in ranked: print(f"{float(s):.3f}", c)

Use Cross-Encoders When

  • Shortlist already retrieved
  • Precision@n dominates UX
  • GPU batching is available

Avoid As Sole Retriever

  • Cannot scan millions online
  • Latency grows with k
  • No independent doc index
Common Misconception

“Cross-encoders replace vector databases.” They complement them. Without a cheap first-stage retriever, you cannot afford joint scoring over the full corpus. Always retrieve then rerank.

Knowledge Check

  1. Short Answer: How does a cross-encoder consume query and document? Answer: Jointly in one forward pass (packed pair).
  2. True/False: Cross-encoders produce a single cached vector per document for ANN. Answer: False.
  3. Multiple Choice: Best production role: (a) sole web-scale search, (b) rerank top-k, (c) replace tokenizers. Answer: (b).
  4. Short Answer: Why is accuracy often higher than bi-encoders? Answer: Full cross-attention between query and document tokens.
  5. True/False: Latency scales with number of candidate pairs scored. Answer: True.
  6. Multiple Choice: Typical first stage before cross-encode: (a) random sample only, (b) dense/sparse/hybrid retrieve, (c) train from scratch each query. Answer: (b).
  7. Short Answer: Name a sentence-transformers class for this. Answer: CrossEncoder.
  8. Short Answer: What separator role appears in BERT-style pairs? Answer: [SEP] (or model-equivalent) between query and document.
  9. Multiple Choice: Output of a reranker is usually: (a) a scalar score, (b) a 12k-d embedding only, (c) a parse tree. Answer: (a).
  10. True/False: You should cross-encode every document in a 10M corpus per query. Answer: False.

Key Takeaways

  • Cross-encoders jointly score query–document pairs with high precision.
  • Use them to rerank shortlists, not to scan entire corpora.
  • Cost is O(k) model passes; keep k modest.
  • Pair with hybrid/dense first-stage retrieval in RAG.
  • Next: Bi Encoder—scalable dual encoders.
Trainer’s Guide

Timing lab: Measure ms/pair vs bi-encoder encode+dot for k=10,50,200.

Discussion: When would an LLM-as-reranker beat a MiniLM cross-encoder?

Recap: Cross-encoders deliver precise pairwise relevance for reranking. Continue with Bi Encoder.