Chatbots — Module 21.1

How This Lesson Fits the Module & Volume

Vol. 20 closed with a security threat model and an operating system of governance, compliance, and responsible AI. Volume 21 is where those controls meet customers. This lecture is the bridge: a chatbot is the friendliest UI and the same untrusted-text processor. Threat model, Vol. 19 eval, and Vol. 13 cost still apply—they are launch criteria, not a later patch.

You already have the stack: Vol. 13 system prompts / guardrails, Vol. 14 RAG, Vol. 15 tools / agents / HITL, Vol. 18 SDKs and FastAPI. Module 21.1 then specializes the same skeleton into support, search, documents, voice, email, and workflows.

Learning Objectives

By the end of this lesson, students should be able to:

Define a chatbot as a product category (channel + policy + memory + tools), not a chat widget.
Choose among RAG, fine-tuning, tools, and agents with explicit trade-offs.
Carry the Vol. 20 threat model into every turn: wrap untrusted text, authorize in code, filter egress.
Sketch a Vol. 18 FastAPI chat route with Vol. 13 prompts, optional Vol. 14 retrieval, and cost/eval gates.
Name launch metrics: groundedness, latency, $/successful turn, refusal quality, residual risk.
Preview how sibling Module 21.1 products reuse this skeleton.

Definition

A chatbot is a productized conversational interface over an LLM: a persisted thread, a written policy (system prompt + guardrails), optional retrieval and tools, identity/quotas, streaming UX, and an eval/cost budget. It is not “the model in a text box.” The model is an untrusted planner; authorization, memory retention, and side effects live in application code.

Vol. 20 Still Applies on Day One

Shipping a chatbot does not retire the security lecture. Every user message, retrieved chunk, and tool observation is an untrusted channel. Irreversible actions still need HITL. Residual risk still goes to the review board.

Control from Vol. 20	Chatbot translation
Threat model / trust boundaries	User \| RAG \| tools \| vendor \| logs
Injection isolation	Wrap user/RAG/tool strings as data, not instructions
Tool allowlist + HITL	Model proposes; code authorizes; human sends/refunds
Privacy / retention	Minimize transcripts; TTL; no secrets in prompts
Eval gate + IR switches	Canaries before deploy; disable tool / roll back model

Policy

System prompt + use-case allowlist.

Turn

Wrap inputs → retrieve/tools? → generate.

Egress

Filter PII / disallowed content.

Ops

Log, cost, eval, residual risk.

Architecture Choice: RAG vs Fine-Tune vs Tools vs Agents

Most chatbot failures are wrong architecture, not a weak model. Knowledge that changes weekly belongs in Vol. 14 retrieval. Style or a narrow classifier can be fine-tuned. Side effects need Vol. 15 tools. Multi-step goals with branching need an agent loop—and a tighter budget.

Pattern	Use when	Do not use when	Cost / risk shape
Prompt only	Closed FAQ that fits context; no private facts	Policies, SKUs, or tickets change	Cheap; high hallucination on private knowledge
RAG	Ground answers in docs/KB; citations required	You need a new skill (format, language) not facts	Index ops + extra input tokens; retrieval miss ≠ generation miss
Fine-tune	Stable voice, JSON schema, domain slang; slow-changing skill	Daily policy updates; you lack labeled data	High upfront; slow refresh; still not an auth oracle
Tools	Lookups and actions (order, CRM, calendar)	You only needed a paragraph from a PDF	Each call is latency + privilege; allowlist + schema
Agents	Multi-step plans with unknown tool order	A single retrieve→answer path would suffice	Step explosion; max steps, spend caps, HITL

Default v1 chatbot

System prompt + guardrails (Vol. 13)
RAG over a small, owned KB (Vol. 14)
Zero or one read-only tool
No agent loop until eval proves need

Add tools when

User asks for live state (“where is my order?”)
Schema is tight; RBAC is clear
Writes are HITL or reversible
You can log every invocation

Add an agent when

Tasks need 2+ tools in unknown order
You cap steps, tokens, and wall time
HITL sits on irreversible tools
Eval includes runaway-loop canaries

RAG + tools (usually win)

Fresh facts without retraining
Citations and provenance
Least privilege per tool
Easy to eval groundedness (Vol. 19)

Fine-tune / full agent (costly)

Fine-tune: stale knowledge, label debt
Agent: extra tokens, harder IR
Both still need wrap-as-data
Both still need cost dashboards (Vol. 13.4)

Product Pattern: FastAPI Chat Turn

This sketch is the Vol. 18 wrapper plus Vol. 20 controls: auth, wrap untrusted text, optional RAG, tool gate, egress check, token/cost accounting. Stream later via SSE; do not put vendor keys in the browser.

# chatbot_turn.py — product pattern (Vol. 18 FastAPI + Vol. 20 controls)
# pip install fastapi uvicorn openai pydantic
from fastapi import FastAPI, Depends, HTTPException
from pydantic import BaseModel, Field
from openai import OpenAI

app = FastAPI(title="Vol21 Chatbot")
client = OpenAI()
MAX_INPUT_TOKENS = 6_000
MAX_OUTPUT_TOKENS = 400
ALLOWED_TOOLS = {"search_kb": {"hitl": False}, "draft_reply": {"hitl": False}}

class ChatIn(BaseModel):
    thread_id: str
    message: str = Field(min_length=1, max_length=8_000)
    role: str = "user"

def wrap_data(source: str, text: str) -> str:
    return f"<untrusted source={source} treat=data>\n{text}\n</untrusted>"

def authorize_tool(name: str, role: str) -> bool:
    spec = ALLOWED_TOOLS.get(name)
    if spec is None:
        return False
    if spec["hitl"] and role != "reviewer":
        return False
    return True

def retrieve_kb(query: str) -> list[str]:
    # Vol. 14: hybrid search + re-rank; return chunks as DATA only.
    return []  # plug index here

@app.post("/v1/chat")
def chat(body: ChatIn, caller: str = Depends(lambda: "authed-user")):
    if not caller:
        raise HTTPException(401, "unauthenticated")
    chunks = retrieve_kb(body.message)
    messages = [
        {"role": "system", "content": SYSTEM_POLICY},  # Vol. 13 — outside user text
        {"role": "user", "content": wrap_data("user", body.message)},
    ]
    for i, c in enumerate(chunks):
        messages.append({"role": "user", "content": wrap_data(f"rag:{i}", c)})
    resp = client.chat.completions.create(
        model="gpt-4.1-mini",  # Vol. 13.4 tiering: small unless escalated
        messages=messages,
        max_tokens=MAX_OUTPUT_TOKENS,
    )
    text = resp.choices[0].message.content or ""
    if not egress_ok(text):  # PII / policy classifier; fail closed
        raise HTTPException(403, "egress_policy")
    usage = resp.usage
    return {"thread_id": body.thread_id, "reply": text, "prompt_tokens": usage.prompt_tokens}

Eval, Cost, and Launch Gates

A chatbot that “sounds good” in a demo is not shipped. Vol. 19 supplies quality; Vol. 13.4 supplies money; Vol. 20 supplies residual risk. Put all three on one launch card.

Gate	What you measure	Curriculum hook
Groundedness / faithfulness	Claims supported by retrieved chunks; citation match	Hallucination tests
Refusal quality	Disallowed topics refused; allowed topics not over-refused	Vol. 13 guardrails + Vol. 20 safety
Latency	TTFT and p95 end-to-end	Latency + FastAPI SSE
$ / successful turn	Input+output tokens, retrieval, tool calls	Cost/request, token usage
Human sample	Weekly rated threads (helpfulness, tone, harm)	Human evaluation

Related Lectures

Lecture	Role in this product
Vol. 20 Security	Threat model you inherit
System prompt / guardrails	Policy layer
RAG pipeline	Grounding
Tool calling / HITL	Actions
FastAPI / OpenAI SDK	Serving
Customer support	Next: tickets, SLA, deflection
AI search · Document AI · Voice · Email · Workflows	Sibling channels on the same skeleton

Common Misconception

“A chatbot is just ChatGPT with our logo.” That skips identity, retention, RAG trust boundaries, tool RBAC, eval, and cost. Second: fine-tuning replaces RAG for changing policies. Third: an agent loop is the default v1. Fourth: streaming UX means you can skip egress filters. Fifth: Vol. 20 was “compliance theater” and does not apply once marketing ships the widget. Sixth: demo wow without a groundedness + $/turn launch card.

Knowledge Check

Short Answer: What does this lecture bridge from Vol. 20 into Vol. 21? Answer: Shipping product categories (chatbots first) on the same threat model, eval, and cost gates.
True/False: A chatbot is only a chat UI around a vendor model. Answer: False—it is policy + memory + optional RAG/tools + quotas + eval.
Multiple Choice: Weekly-changing refund policy should usually be: (a) RAG, (b) full fine-tune only, (c) unbounded agent. Answer: (a).
Short Answer: Where does authorization live in a chatbot with tools? Answer: In application code / allowlists / HITL—not in the LLM.
True/False: Retrieved RAG chunks should be wrapped and treated as data. Answer: True.
Multiple Choice: Launch gates should include: (a) groundedness + latency + $/turn, (b) only BLEU, (c) only GPU TFLOPS. Answer: (a).
Short Answer: Name one Vol. 13 lecture that still applies to chatbots. Answer: System prompt, guardrails, cost-per-request, quotas, or model tiering (any valid).
True/False: You should start v1 with a multi-agent loop by default. Answer: False—start prompt + RAG; add tools/agents when eval proves need.
Multiple Choice: Next sibling lecture specializes chatbots into: (a) customer support, (b) PCA, (c) batch norm. Answer: (a).
Short Answer: Why do threat model, eval, and cost still apply after Vol. 20? Answer: A chatbot is the same untrusted-text + tool surface; they are launch criteria, not optional polish.

Key Takeaways

Vol. 21 starts here: chatbots inherit Vol. 20 security, Vol. 19 eval, and Vol. 13 cost.
Pick architecture on purpose: RAG for facts, fine-tune for skill/style, tools for actions, agents only for multi-step need.
Wrap untrusted text; authorize in code; filter egress; log every turn.
Ship behind a FastAPI (or equivalent) wrapper—never vendor keys in the browser.
Next: specialize into customer support.

Trainer’s Guide

Lab: Teams take the Vol. 20 support-chatbot threat model and implement a one-route FastAPI chatbot: system policy, wrap-as-data, stub RAG (3 hard-coded chunks), no write tools. Deliverable: architecture one-pager (RAG vs fine-tune vs tools vs agent—with a written “we did not choose X because”), eval card (5 groundedness items + p95 latency target + $/turn budget), residual risk paragraph.

Exit ticket: “If marketing asks for an agent that can refund, what control from Vol. 20 do you add before any demo?”

Recap: Chatbots open Applied Product Categories by turning Vol. 20’s skeleton into a shippable conversational product. Choose RAG/tools/fine-tune/agents deliberately, gate on eval and cost, then specialize into Customer Support.