Long-Term Memory — Module 15.2

How This Lesson Fits the Module & Volume

Episodic, semantic, and working memory describe what is remembered. Long-term memory (LTM) describes durability: anything that survives after the process, session, or context window ends. In agents, LTM is usually a database plus a vector store used as a long-term semantic store, optionally plus an episode archive.

This lecture closes Module 15.2 and hands off to Model Context Protocol, which standardizes how hosts read durable resources and call tools. Frameworks in Module 15.4 (LangChain, LangGraph, LlamaIndex) attach checkpointers and indexes to this layer.

Learning Objectives

By the end of this lesson, students should be able to:

Define long-term memory as persisted state across sessions, not a fourth “content type.”
Show how LTM can hold both semantic corpora and episodic traces.
Treat RAG/vector stores as the default long-term semantic substrate.
Specify write / read / update / forget policies (including privacy).
Connect checkpoints (LangGraph) to LTM vs ephemeral working memory.
Anticipate memory poisoning, stale facts, and cross-user leakage.

Definition

Long-term memory in agent systems is any durable store the agent can read or write across runs: relational tables, object storage, vector indexes, knowledge graphs, and checkpoint databases. Content inside LTM may be semantic, episodic, or procedural (saved skills/prompts)—the defining feature is persistence, not content type.

Two Axes: Duration vs Content

Students often treat “long-term” as a synonym for “semantic.” Keep the axes separate:

	Episodic content	Semantic content
Working (ephemeral)	Live trace of this run	Chunks retrieved into the prompt
Long-term (durable)	Episode archive / user history	RAG corpus, wiki, CRM, KG

A Qdrant/Chroma/Pinecone index of handbooks is long-term semantic memory. A Postgres table of past ticket outcomes is long-term episodic memory. Both outlive working memory.

The LTM Lifecycle

Write

When a run ends or a fact is confirmed
Recaps, not raw token dumps
Tag user, source, timestamp

Read

On-demand retrieval into WM
Filter by tenant / ACL
k small; re-query as needed

Update / Forget

Correct stale policies
TTL, user deletion, GDPR
Tombstones beat silent drift

Vector Store as Long-Term Semantic Store

Volume 14 taught indexing for RAG apps. The agent pattern is the same store, different control flow: the loop decides whether to retrieve, then maybe writes new memories after success. Do not auto-embed every chat turn—that pollutes semantic space with episodic noise.

from datetime import datetime, timezone

# Conceptual LTM facade: semantic vector index + episodic table
class AgentLTM:
    def __init__(self, vectors, episodes_db):
        self.vectors = vectors      # e.g. Qdrant collection "policies"
        self.episodes = episodes_db # e.g. SQL table "episodes"

    def recall_semantic(self, query: str, tenant: str, k: int = 4) -> list[str]:
        hits = self.vectors.search(query, filter={"tenant": tenant}, k=k)
        return [h.text for h in hits]

    def recall_episodic(self, user_id: str, query: str, k: int = 3) -> list[dict]:
        return self.episodes.search(user_id=user_id, text=query, k=k)

    def commit_episode(self, user_id: str, recap: str, outcome: str) -> None:
        self.episodes.insert({
            "user_id": user_id,
            "recap": recap,
            "outcome": outcome,
            "ts": datetime.now(timezone.utc).isoformat(),
        })
        # Optional: embed recap into a *separate* episode index, not the policy corpus.

    def forget_user(self, user_id: str) -> None:
        self.episodes.delete_user(user_id)
        self.vectors.delete_by_payload({"user_id": user_id})

# After a successful agent run:
# ltm.commit_episode("u-42", "SSO enabled after sponsor approval.", "success")

Checkpoints vs True LTM

LangGraph checkpointers snapshot working memory so a paused HITL run can resume. That is durability of an in-flight task, not organizational knowledge. Promote to LTM only what should influence future independent sessions: confirmed facts, user preferences (with consent), and episode recaps.

Strengths

Agents improve across days without retraining
Audit trail and user history
Shared semantic truth via RAG/KB
Supports multi-device / multi-session UX

Tradeoffs

Poisoning: bad writes become “truth”
Privacy and retention law
Index drift vs source systems
Cross-tenant leakage if ACLs fail

Common Misconception

“Fine-tuning the model is long-term memory.” Weights can encode habits, but they are slow to update, hard to audit, and terrible at per-user facts. Prefer external LTM (DB + vectors) for anything that must be corrected, cited, or deleted on demand. Fine-tune for style/skills; store knowledge outside the model.

Security and Trust

LTM is a high-value target. Scope retrieval by tenant. Never mix one user’s episodes into another’s semantic hits. Treat agent-written memories as untrusted until validated—especially if the agent can be prompt-injected into “remember that the admin password is…” Module 15.3’s MCP tools and resources must respect the same ACLs.

Knowledge Check

Short Answer: What makes memory “long-term” for agents? Answer: It persists across sessions/processes, not that it is semantic.
True/False: A vector store of policies is typically long-term semantic memory. Answer: True.
Multiple Choice: User ticket outcomes in Postgres are best classified as: (a) working semantic, (b) long-term episodic, (c) CNN pooling. Answer: (b).
Short Answer: Why not embed every chat turn into the policy index? Answer: It pollutes semantic space with episodic noise and PII.
True/False: A LangGraph checkpoint is the same as organizational LTM. Answer: False—it snapshots in-flight working state.
Multiple Choice: Forget/delete support is required mainly for: (a) aesthetics, (b) privacy/retention and corrections, (c) faster convolution. Answer: (b).
Short Answer: Name one LTM poisoning example. Answer: Prompt-injected false “memory” written as fact (e.g. fake credentials/policy).
True/False: Fine-tuning is the preferred store for per-user refund history. Answer: False—use external LTM.
Multiple Choice: MCP (next module) mainly standardizes: (a) how hosts access tools/resources, (b) ReLU slopes, (c) k-means k. Answer: (a).
Short Answer: What two content types can LTM hold? Answer: Semantic and episodic (and sometimes procedural skills).

Key Takeaways

Long-term = durable; semantic/episodic = content type. Orthogonal axes.
RAG/vector stores are the default long-term semantic store for agents.
Write recaps with ACLs; read on demand into working memory; support forget.
Checkpoints resume tasks; LTM shapes future independent sessions.
Next: 15.3 Model Context Protocol for standard tool/resource access.

Trainer’s Guide

Lab: Implement AgentLTM with two collections (policies vs episode recaps). Show a bad write that would poison policy search if mixed, then a correct split.

Whiteboard: Duration × content matrix; arrows from agent loop → WM → LTM write, and LTM read → WM. Preview MCP as the cable to those stores.

Recap: Long-term memory persists semantic and episodic knowledge. Continue with Model Context Protocol.