LlamaIndex — Module 15.4

How This Lesson Fits the Module & Volume

Vol. 14.3 LlamaIndex is the data-centric RAG path: load → index → query engine. This lecture is the knowledge agent: ReAct / function-calling agents that treat query engines and retrievers as tools, mix them with APIs/MCP, and write back into long-term semantic memory.

If your product is retrieval quality, start here. If your product is long-running HITL workflow, prefer LangGraph. If it is role-based teams, see CrewAI.

Learning Objectives

By the end of this lesson, students should be able to:

Contrast LlamaIndex query engines (14.3) with LlamaIndex agents (15.4).
Wrap a VectorStoreIndex as a tool inside a ReAct / FunctionAgent loop.
Combine semantic retrieval with non-RAG tools (tickets, MCP resources).
Avoid stuffing the whole index into working memory.
Choose LlamaIndex vs LangChain vs LangGraph for an agentic RAG product.
Apply citation discipline so semantic hits stay auditable.

Definition

A LlamaIndex agent is an LLM controller that can call tools—especially query engines, retrievers, and data connectors—in a loop to answer or act over private data. The index remains the center of gravity; agency is how the model decides when and how to query it versus other tools.

Query Engine vs Knowledge Agent

	Vol. 14.3 query engine	This lecture (agent)
Flow	One retrieve + synthesize	Multi-step tool loop
Tools	Implicit retriever only	Many: indexes, APIs, MCP, SQL
When	Single grounded question	“Look up policy, then open ticket, then recap”
Memory	Index = semantic LTM	Index + WM trace + optional episodes
Risk	Wrong chunks	Wrong chunks and runaway tools

Agentic RAG Pattern

Index tool

query_engine.query(q)
Long-term semantic store
Return citations + snippets

Systems tools

Tickets, CRM, MCP resources
Grounded IDs after retrieval
HITL on writes

Router

Multi-index / multi-engine
Pick policy vs HR vs code
Still one agent loop

FunctionAgent Sketch (Index as a Tool)

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.llms.openai import OpenAI
from llama_index.core.tools import QueryEngineTool, FunctionTool
from llama_index.core.agent import FunctionAgent

Settings.llm = OpenAI(model="gpt-4o-mini")
index = VectorStoreIndex.from_documents(SimpleDirectoryReader("./policies").load_data())
qe = index.as_query_engine(similarity_top_k=4)

policy_tool = QueryEngineTool.from_defaults(
    query_engine=qe,
    name="policy_search",
    description="Long-term semantic memory: company policies. Cite section titles.",
)

def lookup_ticket(ticket_id: str) -> str:
    """Read one support ticket (system of record, not the wiki)."""
    return f"#{ticket_id} contractor SSO blocked: not in okta-admins"

ticket_tool = FunctionTool.from_defaults(fn=lookup_ticket)

agent = FunctionAgent(
    tools=[policy_tool, ticket_tool],
    llm=Settings.llm,
    system_prompt=(
        "You are a support agent. Use policy_search for rules, "
        "lookup_ticket for case facts. Do not invent policy."
    ),
)

response = agent.run("Ticket 881: can we enable contractor SSO? Cite policy.")
print(response)

Working Memory Discipline

LlamaIndex will happily retrieve verbose nodes. Cap similarity_top_k, ask the tool to return snippets + citations, and compact the agent chat—same 15.2 rules. Do not auto-index every agent utterance into the policy corpus (episodic pollution). For multi-day HITL, export the loop into LangGraph rather than stretching FunctionAgent.

Pick LlamaIndex agents when

Private data / connectors are the hard part
You need many index types (vector, KG, SQL)
Agentic RAG > general app glue
Query quality is the product KPI

Pick something else when

Linear RAG only → Vol. 14.3 query engine
HITL graphs → LangGraph
Role crews → CrewAI
Conversation swarms → AutoGen

Common Misconception

“An agent with a query engine is automatically better than a query engine.” Extra loop steps add cost and failure modes. If the user asks one factual question over docs, use the Vol. 14.3 query engine. Add an agent only when the task branches: compare sources, call a live system, or decide which index to hit.

Knowledge Check

Short Answer: What is LlamaIndex’s center of gravity vs LangChain? Answer: Indexes/data connectors vs general orchestration glue.
True/False: Vol. 14.3 already covered load → index → query; this lecture is agentic use of those indexes. Answer: True.
Multiple Choice: A QueryEngineTool is mainly: (a) long-term semantic access, (b) stdio MCP host, (c) CNN filter. Answer: (a).
Short Answer: When should you not wrap RAG in an agent? Answer: Single-shot grounded Q&A with no extra tools/branches.
True/False: You should embed every agent chat turn into the policy index. Answer: False—that pollutes semantic LTM.
Multiple Choice: Durable HITL ticket workflows belong in: (a) FunctionAgent only, (b) LangGraph (possibly calling LlamaIndex tools), (c) pooling. Answer: (b).
Short Answer: Name one non-index tool a knowledge agent might call. Answer: Ticket/CRM lookup, MCP resource read, SQL, etc.
True/False: Citations still matter in agentic RAG. Answer: True.
Multiple Choice: Multi-index routing still runs inside: (a) one agent loop, (b) k-means, (c) Volume 05 only. Answer: (a).
Short Answer: Which next lecture covers role-based multi-agent crews? Answer: CrewAI.

Key Takeaways

Vol. 14.3 = query engines; Vol. 15.4 = agents that use those engines as tools.
Agentic RAG shines when retrieval must mix with other actions.
Keep k small, cite sources, don’t pollute the semantic index.
Hand off HITL durability to LangGraph when needed.
Continue with CrewAI.

Trainer’s Guide

Lab: Same question via query_engine vs FunctionAgent+ticket tool. Compare latency, tokens, and whether the extra hop was justified.

Whiteboard: Semantic LTM (index) vs systems of record vs episodic recaps. Draw MCP resources/read as an alternate semantic pipe.

Recap: LlamaIndex agents are knowledge agents over your indexes. Continue with CrewAI.