Speech-to-Text — Module 16.1

How This Lesson Fits the Module & Volume

Speech named the modality. Speech-to-text (STT / ASR) is the core observe capability: waveform → words. This lecture goes deep on the task (WER, diarization, streaming, punctuation) without becoming the vendor catalog. Products—Whisper, Deepgram, AssemblyAI, Google, Azure, Amazon, ElevenLabs STT—are Module 16.2. Pair with TTS for the full voice loop and real-time AI for live calls.

Learning Objectives

By the end of this lesson, students should be able to:

Define STT/ASR and its outputs (text, timestamps, confidence, speakers).
Compute and interpret WER / CER on a domain test set.
Choose batch vs streaming STT for a product.
Explain diarization, language ID, and punctuation as add-on tasks.
Call an STT API in Python and store transcripts for RAG/agents.
Know that 16.2 is where you compare Whisper vs cloud STT vendors.

Definition

Speech-to-text (automatic speech recognition, ASR) maps an audio waveform to a token sequence—usually orthographic text—optionally with word timestamps, confidence, language, and speaker labels. It is not audio event tagging (audio) and not TTS.

Outputs That Matter

Field	Why agents need it
Transcript	Working memory / RAG chunk
Timestamps	Jump-to in video, captions
Confidence	HITL when low; don’t tool-call blindly
Speaker labels	Meetings; 16.2 diarization
Language ID	Route to the right LLM / glossary

Batch vs Streaming

Batch / file STT

Podcasts, voicemail, lectures
Best accuracy, optional alignment
Whisper-class models excel here

Streaming STT

Live captions, voice agents
Partials + endpointing
16.2 streaming

Multimodal video

STT soundtrack + keyframes
Video understanding
OCR if burned-in captions

Eval: WER Without Lying to Yourself

Word Error Rate = (substitutions + insertions + deletions) / reference words. A 5% WER on LibriSpeech can be 25% on your call-center jargon. Always measure on in-domain audio with a glossary (product names, acronyms).

# Batch STT + simple WER (jiwer). Swap model in 16.2.
from openai import OpenAI
from jiwer import wer

client = OpenAI()

def transcribe(path: str) -> str:
    with open(path, "rb") as f:
        r = client.audio.transcriptions.create(
            model="whisper-1",
            file=f,
            response_format="verbose_json",
            timestamp_granularities=["word"],
        )
    return r.text

def eval_clip(path: str, reference: str) -> dict:
    hyp = transcribe(path)
    return {"hyp": hyp, "wer": wer(reference.lower(), hyp.lower())}

# Agent observe(): store hyp + words[].start, not the WAV, in episodic/semantic memory.
# For live calls use a streaming vendor (Deepgram / Google / Azure) from 16.2.

Design Rules

Do

Domain test set + glossary / biasing
Denoise / VAD when noise dominates
Diarize meetings; keep speaker turns

Don’t

Use batch STT for barge-in voice agents
Trust STT on non-speech (use audio tags)
Skip PII redaction before logs/RAG

Common Misconception

“STT = Whisper.” Whisper is an excellent batch multilingual model in 16.2. Streaming call centers often pick Deepgram, Google, or Azure for latency and endpointing. The capability (this lecture) is identical: audio in, transcript out. Pick the product after you know batch vs stream, language, and WER on your data.

Knowledge Check

Short Answer: What does STT map, in one line? Answer: Waveform (speech audio) → text (± timestamps/speakers).
True/False: Module 16.1 is the place to memorize every Azure Speech SKU. Answer: False—that is 16.2.
Multiple Choice: WER counts: (a) subs+ins+del over reference words, (b) CLIP cosine, (c) FID. Answer: (a).
Short Answer: Why is in-domain WER required? Answer: Benchmark WER may not match jargon, accents, or channel noise.
True/False: Diarization answers “who spoke when,” not just what was said. Answer: True.
Multiple Choice: Live voice agents need: (a) streaming STT, (b) only overnight batch Whisper, (c) OCR. Answer: (a).
Short Answer: Where should transcripts go in the Vol. 15 memory model? Answer: Working memory now; recaps / chunks into LTM/RAG—not raw audio dumps.
True/False: STT is the right tool for siren detection. Answer: False—audio event tagging.
Multiple Choice: Whisper’s catalog lecture is: (a) 16.2, (b) 16.4, (c) Vol. 09. Answer: (a).
Short Answer: Inverse capability of STT? Answer: Text-to-speech.

Key Takeaways

STT is the speech→text observe capability for agents and video.
Measure in-domain WER; keep timestamps, speakers, confidence.
Batch vs streaming is the first product fork—then open 16.2.
Do not confuse STT with audio tagging or with TTS.
Next: Text-to-speech.

Trainer’s Guide

Lab: Same 10 in-domain clips through (1) local tiny Whisper and (2) one cloud STT. Compute WER; discuss streaming needs even if cloud WER wins.

Preview 16.2: Walk the topic list (Whisper → Deepgram → … → real-time transcription) as a shopping list, not today’s deep dive.

Recap: STT turns speech into text; 16.2 names the models. Continue with Text-to-speech.