Retrieval Augmented Generation (RAG) — Module 14.1

How This Lesson Fits the Module & Volume

Vol. 13 ended on cost-per-request optimization and prompting craft: you can make calls cheaper and clearer, but a frozen LLM still cannot know your private docs, today’s policies, or last week’s tickets. RAG is the next lever—retrieve evidence at query time and ground generation in that context.

Module 14.1 builds the vocabulary: embeddings, chunking, vector search, hybrid retrieval, re-ranking, and indexing. Vol. 12 foundations—dense, sparse, hybrid embeddings, plus bi- and cross-encoders—become the retrieval stack behind every RAG answer.

Learning Objectives

By the end of this lesson, students should be able to:

Define RAG and contrast it with pure prompting and fine-tuning for knowledge.
Sketch the retrieve → augment → generate loop and name each stage’s job.
Explain how retrieved chunks change token economics and grounding quality.
List failure modes: bad retrieval, context stuffing, and uncited hallucination.
Connect Vol. 12 encoders to RAG’s first-stage search and optional re-rank.
Preview the Module 14.1 topic map from embeddings through indexing.

Definition

Retrieval-Augmented Generation (RAG) is a pattern where a system retrieves relevant documents (or chunks) from an external knowledge store for a user query, inserts that evidence into the LLM prompt (or tool context), and then generates an answer conditioned on both the query and the retrieved text.

Why RAG After Cost & Prompting?

Prompting improves how the model uses what it already “knows.” Caching and tiering improve $/success. Neither updates private facts. Fine-tuning can bake in style or narrow skills, but is slow and expensive for daily-changing corpora. RAG keeps the model general and swaps in fresh evidence per request—at the cost of retrieval latency and extra input tokens (exactly the pack_budget / top-k tension Vol. 13 flagged).

Approach	Updates knowledge via	Typical cost pattern
Pure prompting	Whatever fits in the prompt	Low infra; high hallucination risk on private facts
Fine-tuning	Training / adapters	High upfront; slow to refresh
RAG	Index + retrieve at query time	Index ops + extra input tokens; stronger grounding

The Core Loop

Retrieve

Embed / search query
Return top-k chunks
Optional filters / hybrid

Augment

Pack evidence in prompt
Cite sources / IDs
Respect token budget

Generate

Answer from context
Refuse if unsupported
Optionally stream

Code: Minimal RAG Sketch

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
chunks = [
    "Refunds are available within 30 days of purchase.",
    "Shipping to EU takes 5–7 business days.",
    "Support hours are 9am–6pm IST, Monday–Friday.",
]
q = "How long do I have to request a refund?"

emb_c = model.encode(chunks, normalize_embeddings=True, convert_to_tensor=True)
emb_q = model.encode(q, normalize_embeddings=True, convert_to_tensor=True)
hits = util.semantic_search(emb_q, emb_c, top_k=2)[0]
context = "\n".join(f"[{i}] {chunks[h['corpus_id']]}" for i, h in enumerate(hits))

prompt = f"""Use ONLY the context. If missing, say you don't know.
Context:
{context}

Question: {q}
Answer:"""
# hand prompt to your LLM client; cite chunk ids in the answer

Strengths

Grounds answers in your corpus
Updates without retraining the LLM
Enables citations and audit trails

Tradeoffs

Retrieval quality bounds answer quality
More input tokens → higher $/request
Index freshness and ops become critical

Common Misconception

“If we add RAG, the model cannot hallucinate.” Wrong. RAG reduces unsupported guessing when retrieval is good and the prompt forces evidence use. Bad chunks, noisy top-k, or soft instructions still produce fluent but false answers—now with fake citations.

Knowledge Check

Short Answer: What three verbs summarize RAG? Answer: Retrieve, augment (prompt), generate.
True/False: RAG replaces the need for clear prompting. Answer: False—prompts still enforce evidence use and format.
Multiple Choice: RAG primarily solves: (a) CSS layout, (b) external/up-to-date knowledge grounding, (c) GPU overclocking. Answer: (b).
Short Answer: How does RAG interact with Vol. 13 cost work? Answer: Retrieved context adds input tokens; top-k and packing affect CPSR.
True/False: Fine-tuning is always cheaper than RAG for daily FAQ updates. Answer: False.
Multiple Choice: First-stage semantic search often uses: (a) a bi-encoder, (b) only CSS, (c) PCA labels. Answer: (a).
Short Answer: Name one RAG failure mode. Answer: Bad retrieval, context overflow, or uncited hallucination (any).
True/False: Cross-encoders are often used to re-rank a small candidate set. Answer: True.
Multiple Choice: Next Module 14.1 topic after this overview: (a) Embedding for retrieval, (b) Vol. 1 only, (c) printers. Answer: (a).
Short Answer: Why cite chunk IDs in the prompt? Answer: So the model (and user) can attribute claims to retrieved evidence.

Key Takeaways

RAG grounds generation in retrieved evidence instead of parametric memory alone.
Quality is gated by retrieval, packing, and instruction discipline.
Token budgets from Vol. 13 still apply—every chunk costs money.
Next: Embedding (for retrieval).

Trainer’s Guide

Lab: Take a 20-doc FAQ; run the MiniLM sketch; compare answers with vs without retrieved context on 10 questions.

Discussion: When would you fine-tune and RAG instead of choosing one?

Recap: RAG bridges prompting/cost control into grounded systems. Continue with Embedding (for retrieval).