Voice Assistants — Module 21.1

How This Lesson Fits the Module & Volume

Chatbots are text turns; voice assistants are the same product over audio: Vol. 16 speech-to-text, NLU/LLM, tools, then TTS. Latency, barge-in, and confirmation replace typing UX. Vol. 20 threat model still applies—transcripts are untrusted text, just noisier.

Reuse Vol. 13 prompts/guardrails, Vol. 14 RAG for spoken FAQs, Vol. 15 tools/HITL for “book that” / “refund that,” Vol. 18 FastAPI + streaming / WebSockets, Vol. 19 latency and human eval. Siblings: support phone trees, search by voice, docs read-back, email dictation into workflows.

Learning Objectives

By the end of this lesson, students should be able to:

Define a voice assistant as STT → policy LLM → tools → TTS with hard latency budgets.
Choose RAG vs fine-tune vs tools vs agents under audio constraints.
Require confirmation (HITL-in-the-ear) for irreversible actions.
Separate ASR error from NLU/LLM error in eval.
Sketch a FastAPI/WebSocket turn with wrap-as-data on transcripts.
Apply Vol. 20: audio + transcripts are untrusted; minimize retention.

Definition

A voice assistant is a conversational product whose primary I/O is speech: automatic speech recognition (ASR/STT) produces a transcript, an application policy + optional RAG/tools plans a response or action, and text-to-speech (TTS) speaks it. Barge-in, timeouts, and confirmation prompts are first-class UX. The LLM still is not the authorization oracle.

Why Voice Changes the Chatbot Skeleton

Users interrupt. ASR mishears order IDs. You cannot show eight citations on a phone call. p95 latency includes STT + retrieve + LLM + TTS. Cost includes audio minutes plus tokens. Privacy includes voiceprints and ambient speech (Vol. 20 privacy).

Text chatbot	Voice assistant
User edits typos	ASR errors become “facts” unless you confirm
Streaming tokens feel fast	TTFT + TTS chunking; target < ~800 ms first audio when possible
Show sources in UI	Speak one answer; offer to text a link
Click to confirm refund	Read-back + explicit yes; else HITL queue
Long RAG context OK-ish	Aggressively tier models and top-k (Vol. 13.4)

Listen

STT + VAD; barge-in.

Understand

Transcript as untrusted data.

Act / retrieve

Tools + RAG; confirm writes.

Speak

TTS stream; log transcript TTL.

Architecture Choice: RAG vs Fine-Tune vs Tools vs Agents

Pattern	Voice fit	Risk / cost
Fine-tune / small NLU	Intents: timer, weather, “speak to human”	Good latency; weak on changing FAQs
RAG	Spoken help center; tiny top-k	Extra STT wait + retrieval; still wrap chunks
Tools	Calendar, order status, smart home	Misheard slot values; confirm IDs out loud
Agents	Rare on a live call	Step explosion kills latency/cost; prefer scripted flows + one LLM turn

v1 voice FAQ

STT → small model + RAG k=3
No write tools
Offer human handoff keyword
TTS short answers

v2 with actions

Slot fill + read-back
“Pay $49 to Acme—yes or no?”
HITL if ASR conf low
Idempotency on device/API

Avoid on-call agents

Multi-hop browse mid-call
Uncapped tool loops
Long chain-of-thought spoken aloud
Fine-tune as policy oracle

Do

Measure WER / ASR conf vs downstream task success
Confirm numbers, names, money
Stream TTS; barge-in cancels speak
Retain audio only as long as policy allows

Don’t

Treat transcript as system prompt
Speak PII the user did not just provide
Use a giant agent because “voice is multimodal”
Skip Vol. 19 latency SLOs

Product Pattern: Voice Turn over WebSocket

Conceptual FastAPI/WebSocket handler: STT already produced text + confidence. Same chatbot controls; extra confirm gate before tools.

# voice_turn.py — after STT (Vol. 16) on a Vol. 18 WebSocket
from pydantic import BaseModel, Field

WRITE_TOOLS = {"place_order", "send_money", "delete_event"}
ASR_HITL = 0.75

class VoiceTurn(BaseModel):
    session_id: str
    transcript: str = Field(max_length=8_000)
    asr_conf: float = Field(ge=0, le=1)
    user_confirmed: bool = False  # second utterance "yes"
    proposed_tool: str | None = None
    tool_args: dict = {}

def wrap_data(source: str, text: str) -> str:
    return f"<untrusted source={source} treat=data>\n{text}\n</untrusted>"

def handle_voice_turn(t: VoiceTurn, retrieve, llm_plan, tts) -> dict:
    if t.asr_conf < ASR_HITL and t.proposed_tool in WRITE_TOOLS:
        return {"speak": "I might have misheard. Please type or say the details again.", "status": "reprompt"}
    chunks = retrieve(t.transcript)[:3]
    messages = [
        {"role": "system", "content": VOICE_POLICY},  # short spoken style; Vol. 13
        {"role": "user", "content": wrap_data("stt", t.transcript)},
    ]
    for i, c in enumerate(chunks):
        messages.append({"role": "user", "content": wrap_data(f"rag:{i}", c)})
    plan = llm_plan(messages)  # {say, tool?, args?}
    tool = plan.get("tool")
    if tool in WRITE_TOOLS and not t.user_confirmed:
        return {"speak": f"Just to confirm: {plan.get('confirm_phrase')}. Say yes to continue.", "status": "need_confirm", "pending": plan}
    if tool and not authorize(tool, t.tool_args or plan.get("args", {})):
        return {"speak": "I can't do that. I can connect you to a person.", "status": "denied"}
    say = plan.get("say", "")
    if not egress_ok(say):
        say = "I can't go into that. Want a human instead?"
    audio_iter = tts(say)  # stream chunks; honor barge-in
    return {"speak": say, "audio": audio_iter, "status": "ok", "tool": tool}

Eval: Split ASR from Dialogue from Action

Layer	What you score	Hook
STT	WER / CER; keyword recall on IDs	Vol. 16 STT
Dialogue	Task success, groundedness, over-refusal	Vol. 19 human eval + hallucination tests
Action safety	Unauthorized writes = 0; confirm coverage	Vol. 15 HITL + Vol. 20
Latency / cost	Time-to-first-audio; audio minutes + tokens	Vol. 19 latency, Vol. 13.4 cost

Related Lectures

Lecture	Role
Speech-to-text / TTS / speech	Audio I/O
Chatbots / Customer support	Same policy brain
HITL / tool calling	Confirm + act
WebSockets / streaming	Realtime transport
AI search · Document AI · Email · Workflows	Sibling products on voice channel

Common Misconception

“Voice is just STT glued onto ChatGPT.” Latency, barge-in, confirmation, and ASR confidence dominate product quality. Second: agents are better on calls because they “think more.” Third: low WER means refunds are safe. Fourth: you can skip wrap-as-data because audio is analog. Fifth: keep all recordings forever for “quality.” Sixth: speaking citations is a substitute for not retrieving.

Knowledge Check

Short Answer: What pipeline defines a voice assistant? Answer: STT → policy LLM (+ RAG/tools) → TTS, with latency and confirmation.
True/False: Transcripts should be wrapped and treated as untrusted data. Answer: True.
Multiple Choice: Irreversible voice actions should: (a) be confirmed out loud / HITL, (b) run immediately for UX, (c) use jailbreak prompts. Answer: (a).
Short Answer: Why avoid multi-step agents on a live call by default? Answer: Latency and cost explode; misheard slots compound; prefer one turn + tools + confirm.
True/False: Low ASR confidence is a reason to block write tools even if the LLM is fluent. Answer: True.
Multiple Choice: Eval should split: (a) ASR vs dialogue vs action safety, (b) only BLEU on TTS, (c) only GPU TFLOPS. Answer: (a).
Short Answer: Name one Vol. 16 lecture this product depends on. Answer: Speech-to-text, TTS, speech, or real-time AI (any valid).
True/False: Voice assistants inherit the Vol. 20 threat model. Answer: True.
Multiple Choice: Spoken help-center answers should usually use: (a) tiny-k RAG, (b) fine-tune weekly policy into weights, (c) unbounded web agents. Answer: (a).
Short Answer: Which sibling lecture is the text analog of this channel? Answer: Chatbots (or customer support).

Key Takeaways

Voice = chatbot skeleton + STT/TTS + latency + confirmation + retention rules.
Prefer small NLU + RAG + allowlisted tools; keep agents off the live call path.
Confirm money/IDs; fail closed on low ASR confidence.
Eval WER separately from task success, faithfulness, and unauthorized writes.
Next: email automation—another async untrusted-text channel.

Trainer’s Guide

Lab (no live mics required): Provide transcripts with ASR confidence and one misheard amount (“forty” vs “fourteen”). Students implement confirm-before-write. Grade: zero unconfirmed writes, wrap-as-data present, spoken answer ≤ 2 sentences, escalate phrase when denied.

Demo option: Wire Vol. 16 STT/TTS stubs and measure time-to-first-audio with a small vs large model (Vol. 13.4 tiering).

Recap: Voice assistants put STT and TTS around the same Vol. 20-hardened chatbot, with stricter latency and confirmation. Next async channel: Email Automation.