Speech — Module 16.1

How This Lesson Fits the Module & Volume

Volume 15 closed with AutoGen and the agent stack: loops, memory, tools, MCP. Those agents still mostly read and write text. Volume 16 adds the other senses—what the agent can hear, see, and generate. This lecture opens Module 16.1 by treating speech as the first multimodal I/O channel: spoken language in, spoken language out.

16.1 teaches modalities and capabilities (what the task is), not vendor catalogs. Product-level STT/TTS systems live in Module 16.2; image models in 16.3; video models in 16.4. After speech you will map vision, video, and audio the same way.

Learning Objectives

By the end of this lesson, students should be able to:

Define speech as a language modality distinct from general audio.
Draw the agent speech loop: mic → STT → LLM/agent → TTS → speaker.
Contrast batch vs streaming speech, and offline vs real-time AI.
Name the related capabilities: STT, TTS, voice cloning, diarization.
Wire a minimal Python STT+TTS round-trip an agent can call as a tool.
Know where Module 16.2 picks specific STT/TTS products (Whisper, Deepgram, ElevenLabs, …).

Definition

Speech is spoken language: a time-series of acoustic features that encode words, speaker identity, prosody, and (often) emotion. As a capability, speech I/O means converting between that waveform and text—or synthesizing a new waveform—so an AI agent can listen and talk without changing its reasoning loop.

Speech vs Audio vs Text

Text is discrete tokens. Speech is a continuous waveform that carries language plus extra channels: who is speaking, how fast, whether they are sarcastic. Audio is broader still—music, alarms, footsteps, engine noise—where there may be no words at all. Do not collapse these three into one API call.

Signal	What it encodes	Typical task	This module
Text	Tokens / semantics	LLM chat, RAG	Vol. 11–15 (already built)
Speech	Language + speaker + prosody	STT, TTS, cloning	This lecture + 16.2
Audio (non-speech)	Events, music, ambience	Tagging, separation	Audio
Vision / video	Pixels + time	See / generate / understand	Vision, Video

The Agent Speech Loop

Keep Volume 15’s loop intact. Speech is just another observation and another action:

Observe

Microphone or call audio
STT → transcript
Optional diarization / language ID

Reason & act

Same agent / tool loop
Transcript sits in working memory
Tools still via MCP / function calling

Respond

LLM text reply
TTS → waveform
Optional cloned voice / streaming

A voice agent that skips STT and “just sends audio to the LLM” is still doing speech recognition inside the model. You still need latency budgets, punctuation, speaker labels, and a transcript for logs and RAG. Treat STT as an explicit stage unless you have a true end-to-end speech-to-speech model and you accept opaque logs.

Capability Map (Not the Catalog)

Capability	Input → output	Where we teach it
Speech-to-text	Waveform → words	16.1 STT → 16.2 Whisper+
Text-to-speech	Words → waveform	16.1 TTS → 16.2 ElevenLabs / PlayHT
Voice cloning	Sample + text → that voice	Voice cloning
Speaker diarization	Mix → “who spoke when”	16.2 Diarization
Real-time / streaming	Chunks with low latency	Real-time AI, 16.2 streaming

Minimal Python Round-Trip

This pattern is what an AutoGen / LangGraph tool actually calls. Swap the client for Whisper, Deepgram, or Azure later—the capability stays the same.

# Conceptual speech I/O for an agent tool (OpenAI-style APIs)
from pathlib import Path
from openai import OpenAI

client = OpenAI()

def listen(wav_path: str) -> str:
    """STT: waveform -> transcript (observe)."""
    with open(wav_path, "rb") as f:
        result = client.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            response_format="verbose_json",  # keep timestamps if you need them
        )
    return result.text.strip()

def speak(text: str, out_path: str = "reply.mp3") -> str:
    """TTS: agent text -> waveform (act)."""
    audio = client.audio.speech.create(
        model="gpt-4o-mini-tts",
        voice="alloy",
        input=text,
    )
    Path(out_path).write_bytes(audio.read())
    return out_path

def voice_turn(wav_path: str, agent_reply_fn) -> str:
    user_text = listen(wav_path)
    reply_text = agent_reply_fn(user_text)   # Vol. 15 loop unchanged
    return speak(reply_text)

# Do not store raw audio in semantic memory; store the transcript (+ optional recap).

Quality, Latency, and Logging

Design for

WER / CER on your accents and jargon
Streaming partials for UX (<300 ms felt latency)
Punctuation, numbers, and PII redaction
Transcript as the system of record

Watch out for

Batch STT on a live call (feels broken)
TTS without barge-in / interrupt
Cloning a voice without consent
Dumping hours of audio into the context window

Common Misconception

“Speech = Whisper.” Whisper is one model in the 16.2 STT catalog. Speech is the modality: STT, TTS, cloning, diarization, and real-time transport. An agent can use Deepgram for streaming STT and ElevenLabs for TTS and still be doing the same speech capability you define here.

Knowledge Check

Short Answer: How does this lecture connect Volume 15 to Volume 16? Answer: Agents stay the same; speech becomes a new observe/act channel (STT in, TTS out).
True/False: Speech and general audio are the same modality. Answer: False—speech is spoken language; audio includes non-speech sound.
Multiple Choice: The observe stage of a voice agent is usually: (a) TTS, (b) STT, (c) image captioning. Answer: (b).
Short Answer: What should go into working/semantic memory—raw WAV or the transcript? Answer: The transcript (plus a recap); not hours of raw audio.
True/False: Module 16.1 is the vendor catalog for Whisper, Deepgram, and ElevenLabs. Answer: False—16.1 is capabilities; 16.2 is the STT/TTS catalog.
Multiple Choice: Voice cloning belongs with: (a) k-means, (b) speech synthesis identity, (c) OCR. Answer: (b).
Short Answer: Name two speech capabilities besides STT and TTS. Answer: Voice cloning, diarization, streaming/real-time, language ID (any two).
True/False: Batch transcription is the right default for a live phone agent. Answer: False—you need streaming / real-time STT.
Multiple Choice: Product-level image generators are covered in: (a) 16.1 only, (b) 16.3, (c) Volume 05. Answer: (b).
Short Answer: Which lecture comes next in 16.1? Answer: Vision.

Key Takeaways

Speech is spoken language I/O for agents—not the same as music/event audio.
Keep the Vol. 15 loop: STT observe, reason/tools, TTS act.
16.1 names capabilities; 16.2 names STT/TTS products.
Log transcripts, not raw audio dumps; respect consent for cloning.
Next: Vision as the seeing modality.

Trainer’s Guide

Lab: Take the Volume 15 SSO / helpdesk agent. Add listen() and speak() tools. Students must show the transcript in logs even if the demo is voice-only.

Whiteboard: Three columns—modality (speech), capability (STT/TTS/clone), product (Whisper / Deepgram / ElevenLabs). Repeat this map for vision → 16.3 and video → 16.4.

Recap: Speech opens Volume 16 by plugging spoken I/O into the agent stack from AutoGen. Continue with Vision.