Semantic Memory — Module 15.2

How This Lesson Fits the Module & Volume

Episodic memory answered “what happened last time?” Semantic memory answers “what is true about the world (or this org)?”—policies, entity facts, product specs, ontologies. Volume 14 already taught RAG and vector stores; here we reuse that stack as the agent’s long-term semantic store, not as a standalone Q&A app.

You will connect this to working memory (what fits in the prompt now) and long-term memory (persistence). Later, MCP resources and LlamaIndex agents expose semantic corpora as tools and indexes.

Learning Objectives

By the end of this lesson, students should be able to:

Define semantic memory as decontextualized facts, concepts, and relations.
Contrast semantic vs episodic vs working memory with agent examples.
Map RAG/vector stores and knowledge graphs onto semantic memory.
Write a retrieve-then-reason step that injects facts into the agent loop.
Explain freshness, authority, and conflict when facts disagree with episodes.
Choose structured KB vs embeddings vs hybrid for a given domain.

Definition

Semantic memory is memory for general knowledge—meanings, facts, and relationships that are not tied to a single personal event. In agents, it is the knowledge base the model consults: documents, tables, embeddings, and graphs that remain valid across many users and sessions.

Facts vs Experiences vs Scratch

Statement	Type	Why
“Refunds require a receipt within 30 days.”	Semantic	Policy fact, not a dated event
“User U-42 was denied a refund on 12 May.”	Episodic	Specific event + outcome
“Current ticket #881; next tool: lookup_okta.”	Working	Live task state in the loop
Embedded employee handbook in Qdrant	Long-term semantic	Persisted RAG corpus

How Agents Implement Semantic Memory

Vector RAG

Chunk + embed + retrieve
Great for prose policies
Fuzzy match, weaker precision

Structured KB

SQL, CRM, knowledge graph
Exact entities and relations
Needs schemas and ETL

Hybrid

Graph/SQL for IDs + RAG for prose
Best for enterprise agents
MCP can expose both

Volume 14’s RAG pipeline is the usual long-term semantic store: retrieve k chunks, stuff them into working memory, then let the reasoner decide. The agent twist is when to retrieve: not only at the start of a query, but mid-loop when a tool result reveals a missing concept (“what is Okta group X?”).

Retrieve-Then-Reason in the Agent Loop

# Semantic memory as a tool the agent can call mid-loop
from typing import Callable

def semantic_lookup(query: str, retriever: Callable[[str], list[str]], k: int = 4) -> str:
    chunks = retriever(query)[:k]
    return "\n---\n".join(chunks) if chunks else "No semantic match."

TOOLS = {
    "semantic_lookup": {
        "description": "Search company knowledge (policies, specs). Not for user history.",
        "fn": lambda q: semantic_lookup(q, retriever=my_vector_retriever),
    },
    "episode_lookup": {
        "description": "Search this user's past runs/outcomes.",
        "fn": lambda q: episode_store.similar(q),
    },
}

def agent_step(goal: str, scratch: list[str]) -> str:
    # Planner chooses a tool; semantic vs episodic is an explicit fork.
    if "policy" in goal.lower() or "what is" in goal.lower():
        fact = TOOLS["semantic_lookup"]["fn"](goal)
        scratch.append(f"SEMANTIC:\n{fact}")
    return scratch[-1]

print(agent_step("What is the 30-day refund policy?", []))

Authority, Freshness, and Conflicts

Semantic stores go stale. An episode from yesterday can contradict a handbook last indexed in January. Production agents need: source tags (policy v3.2 vs chat recap), timestamps, and a precedence rule—usually authoritative semantic docs beat anecdotal episodes, unless the episode records a granted exception. Never silently merge them in working memory without labels.

Strengths

Grounds agents in org truth
Reusable across users
RAG/KG tooling is mature
Complements (does not replace) episodes

Tradeoffs

Retrieval noise / wrong chunk
Stale indexes vs live systems
Over-retrieval fills working memory
Hallucinated “facts” if retrieval fails

Common Misconception

“The LLM’s pretrained weights are the agent’s semantic memory.” Weights are frozen world priors. Agent semantic memory is your corpus and systems of record—retrieved at runtime. Treating parametric knowledge as the KB causes silent, un-auditable errors on private or changing facts.

Knowledge Check

Short Answer: Define semantic memory in one sentence for agents. Answer: General facts/concepts/relations not bound to one event, usually a KB or RAG corpus.
True/False: “User was refunded yesterday” is semantic memory. Answer: False—it is episodic.
Multiple Choice: The typical long-term semantic store in modern agents is: (a) CNN filters, (b) vector RAG / KB, (c) GPU VRAM only. Answer: (b).
Short Answer: When should an agent call semantic_lookup mid-loop? Answer: When a missing concept/policy is needed to continue reasoning or tool use.
True/False: Pretrained LLM weights replace a company knowledge base. Answer: False.
Multiple Choice: If a handbook and a recent episode disagree, prefer: (a) unlabeled blend, (b) labeled sources + authority rule, (c) always the episode. Answer: (b).
Short Answer: Name one structured alternative to vector RAG for semantic memory. Answer: SQL/CRM, knowledge graph, or similar system of record.
True/False: Semantic chunks belong in working memory only after retrieval (or selection). Answer: True—do not dump the whole corpus.
Multiple Choice: LlamaIndex (Vol. 14.3 / 15.4) is especially strong at: (a) indexing/query for semantic data, (b) TCP congestion, (c) image pooling. Answer: (a).
Short Answer: Which lecture covers the live, capacity-limited scratchpad? Answer: Working Memory.

Key Takeaways

Semantic memory is decontextualized knowledge; episodes are dated experiences.
RAG/vector stores and KBs are the standard long-term semantic substrate.
Retrieve on demand into working memory; label sources and recency.
Parametric LLM knowledge is not your org’s semantic memory.
Continue with Working Memory.

Trainer’s Guide

Lab: Give students a tiny policy doc + three fake user episodes. Ask the agent to answer a refund question; require it to cite semantic vs episodic sources separately.

Whiteboard: Draw retrieve-then-reason inside the agent loop; mark context-budget limits heading into working memory.

Recap: Semantic memory is the agent’s fact store—often RAG. Continue with Working Memory.