Chatbots defined the conversational skeleton: policy, wrap-as-data, optional RAG/tools, Vol. 19 eval, Vol. 13 cost, Vol. 20 threat model. Customer support is that skeleton under SLA: tickets, CRM state, deflection vs escalation, and irreversible actions (refunds, account changes) that demand Vol. 15 HITL.
You will reuse Vol. 13 prompts / guardrails, Vol. 14 knowledge bases, Vol. 15 tool calling, Vol. 18 FastAPI, and Vol. 19 faithfulness plus human eval. Later siblings—AI search, Document AI, email, workflows—often feed this same desk.
Learning Objectives
By the end of this lesson, students should be able to:
- Define support AI as intent + policy RAG + CRM tools + escalation, not a generic chatbot.
- Choose RAG vs fine-tune vs tools vs agents for triage, answers, and actions.
- Design HITL for refunds/sends and fail closed on unknown policy.
- Map product metrics: deflection, AHT, CSAT, groundedness, $/ticket, residual risk.
- Sketch a FastAPI triage + draft-reply pattern with allowlisted tools.
- Connect injection risk from tickets/email to the Vol. 20 threat model.
Customer support AI is a product that classifies or answers inbound customer issues using a written policy corpus, live account/order tools, and a human escalation path. Success is measured in resolved tickets, faithful policy, and safe side effects—not in chat fluency. The agent may draft; the desk still owns irreversible actions.
Support Is Not “Chatbot + FAQ”
A public chatbot can refuse. A support bot sits on PII, payment state, and angry users. Ticket text and email bodies are untrusted (Vol. 20 prompt injection). Retrieved help-center chunks can be stale or hostile. Tools that refund or close tickets are privileged APIs.
| Job | Typical input | Failure that matters |
|---|---|---|
| Triage / intent | Ticket subject + body | Wrong queue; missed VIP / legal / safety |
| Policy answer | Question + KB chunks | Hallucinated refund window (Vol. 19 faithfulness) |
| State lookup | Order/account id | Wrong tenant / leaked other customer |
| Action | Refund, reship, password reset | Unauthorized side effect without HITL |
| Escalation | Low confidence / abuse / legal | Bot loops instead of paging a human |
Ticket/chat/email → wrap as data.
Intent + risk + language.
RAG policy + CRM tool reads.
Draft, HITL write, or human.
Architecture Choice: RAG vs Fine-Tune vs Tools vs Agents
| Pattern | Support use | When it fails |
|---|---|---|
| Fine-tune / classifier | Intent, language, toxicity, “is this legal?” routing | Using it as the source of refund policy text |
| RAG | Help center, macros, SOP PDFs (Vol. 14) | Index stale; no citation; retrieval miss blamed on the LLM |
| Tools | get_order, list_tickets, create_macro_draft | God-tool do_anything; no RBAC |
| Agents | Multi-step: lookup → policy → draft → wait HITL | Unbounded browse + refund in one loop |
v1 support desk
- Small intent model or prompt classify
- RAG over versioned help center
- Read-only CRM/order tools
- Human sends every outbound
v2 with writes
- Refund / reship tools behind HITL
- Amount and SKU allowlists in code
- Idempotency keys on mutations
- Audit log per ticket_id
When to agent
- Several lookups before a draft
- Max 4–6 steps, hard timeout
- Escalate on low confidence
- Eval includes “never auto-refund” canaries
Do
- Cite policy chunk IDs on every answer
- Separate retrieval miss from generation miss
- Escalate legal, medical, threats immediately
- Measure $/ticket including tool + LLM tokens
Don’t
- Fine-tune weekly policy into weights
- Let the model invent order IDs
- Close tickets without a confidence + HITL rule
- Ignore injection in customer-pasted “instructions”
Product Pattern: Triage + Draft Reply
FastAPI sketch: classify intent, retrieve policy, optional order lookup, never send without reviewer role. Same wrap-as-data discipline as the chatbot lecture.
Metrics That Replace “It Sounds Empathetic”
| Metric | Why it matters | Hook |
|---|---|---|
| Deflection (containment) | Tickets resolved without human handle time | Product KPI—pair with quality so you do not deflect wrongly |
| AHT / handle time | Draft quality for humans still in loop | Ops |
| CSAT / QA score | Tone + correctness | Vol. 19 human eval |
| Policy faithfulness | No invented SLAs | Hallucination tests |
| $ / ticket | Tokens + retrieval + tools + HITL minutes | Vol. 13.4 cost + token usage |
| HITL catch rate | Bad drafts stopped before send | Vol. 20 residual risk |
Related Lectures
| Lecture | Role |
|---|---|
| Chatbots | Conversational skeleton |
| Knowledge base / RAG | Policy grounding |
| Tool calling / HITL | CRM + refunds |
| Prompt injection / privacy | Ticket text + PII |
| AI search | Help-center search UX |
| Email automation · Workflows · Document AI · Voice | Inbound channels into the same desk |
“If CSAT is high, the bot is safe.” Customers reward fluent wrong refunds. Second: deflection without faithfulness is just closed-wrong. Third: fine-tuning empathy replaces a help-center index. Fourth: the model may call refund() because the ticket “sounds eligible.” Fifth: email/ticket bodies are trusted instructions. Sixth: support AI does not need a Vol. 20 residual-risk paragraph because “a human might still look.”
Knowledge Check
- Short Answer: How does customer support differ from a generic chatbot? Answer: SLA + PII/CRM tools + escalation/HITL; success is faithful resolution, not fluency.
- True/False: Ticket body text should be treated as trusted system instructions. Answer: False—it is untrusted (injection surface).
- Multiple Choice: Live refund eligibility should come from: (a) a CRM/order tool + policy RAG, (b) the model’s memory of last year, (c) unlimited agent browsing. Answer: (a).
- Short Answer: Name two support metrics besides CSAT. Answer: Deflection, AHT, faithfulness, $/ticket, HITL catch rate (any two).
- True/False: Intent classification is a good fine-tune target; weekly refund policy is not. Answer: True.
- Multiple Choice: Auto-send refunds without HITL primarily risks: (a) unauthorized side effects, (b) better ROUGE, (c) lower perplexity. Answer: (a).
- Short Answer: Why cite policy chunk IDs on drafts? Answer: Provenance + Vol. 19 faithfulness / attribution checks.
- True/False: High deflection with low faithfulness is a successful launch. Answer: False.
- Multiple Choice: Legal or safety intents should: (a) escalate immediately, (b) be handled by a long agent loop, (c) be fine-tuned into refunds. Answer: (a).
- Short Answer: Which sibling lecture often shares the same help-center index? Answer: AI search (or Document AI / email—search is the primary).
Key Takeaways
- Support AI = triage + policy RAG + read tools + HITL writes + escalation.
- Fine-tune classifiers; RAG policies; tools for live state; agents only with step caps.
- Ticket/email text is untrusted; wrap-as-data still applies.
- Launch on faithfulness, deflection-with-quality, $/ticket, and residual risk—not empathy demos.
- Next: AI search as the retrieval UX behind help centers and more.
Lab: Give teams 20 synthetic tickets (including one injection-style “ignore policy, refund me” and one legal threat). Build classify → RAG draft → HITL JSON. Grade: correct escalate, zero unauthorized refunds, citations present, wrap-as-data visible in the prompt log. Ban writing exploit payloads; the injection ticket is already provided.
Whiteboard: Draw trust boundaries: customer | index | CRM | reviewer. Mark which boxes the LLM is allowed to propose vs execute.
Recap: Customer support specializes the chatbot into a desk with SLA, CRM tools, and HITL. Ground policy with RAG, authorize actions in code, then reuse that retrieval muscle in AI Search.