Real-Time Transcription — Module 16.2

How This Lesson Fits the Module & Volume

This lecture is the operating system for everything in 16.2: when to stream vs batch, how partials work, where latency hides, and how PII and diarization behave under a live SLO. You already met streaming APIs in Deepgram, Google, Azure, and Transcribe, and learned why Whisper is a poor live engine.

It also closes the module before Volume 16.3’s image catalog (DALL·E). Tie back to real-time AI and the voice loop with ElevenLabs / PlayHT TTS. Agents that consume live transcripts live in Volume 15.

Learning Objectives

By the end of this lesson, students should be able to:

Contrast batch vs streaming STT on latency, WER posture, cost, and operational complexity.
Define interim vs final transcripts, endpointing, and time-to-first-word.
Sketch a streaming client (chunked PCM + event handler) without inventing vendor internals.
Place diarization, PII redaction, and language ID on the live path vs the offline re-decode path.
Choose a vendor mix for captions, voice agents, and compliance archives.
List failure modes: jitter, packet loss, barge-in, overlapping speech, runaway partials.

Definition

Real-time transcription (streaming STT) converts audio to text with bounded delay while speech is still happening. The recognizer emits interim (unstable) hypotheses for UI, then final utterances after endpointing. It is a different product contract from batch ASR, which waits for a closed file (or long chunk) and typically yields better punctuation, diarization, and WER at the cost of latency.

Batch vs Streaming — The Decision Table

Dimension	Streaming (live)	Batch / offline
Latency	Hundreds of ms to low seconds to first useful text	Seconds to minutes after hang-up / upload
Accuracy posture	Often slightly worse: limited future context, aggressive endpointing	Usually best WER, punctuation, diarization
Languages	Subset of batch lists on some vendors—verify streaming locale support	Widest model/locale matrix
Pricing posture	Often a distinct per-minute live SKU (sometimes pricier)	Pre-recorded / job SKUs; GPU time if self-host Whisper
On-prem vs cloud	Most live APIs are cloud WS/gRPC; on-prem needs containers or self-host streaming stacks	Whisper/pyannote on-prem is easiest
Diarization	Coarse live labels if offered	Preferred for DER-sensitive archives
PII	Redact finals before logs/LLM; partials still leak if you log them	Redact full transcript + optional redacted audio
Typical pick	Deepgram, Azure SDK, Google streaming, AWS streaming	Whisper, AssemblyAI jobs, GCS/S3 batch, hyperscaler long-running

Latency Budget (Where Milliseconds Go)

Capture

Mic / PSTN / WebRTC jitter buffer
Frame size 20–100 ms
Resample to 16 kHz mono PCM often

Recognize

Time to first partial
Endpointing wait
Model + queue on vendor side

Act

LLM time (if agent)
TTS TTFB (ElevenLabs/PlayHT/Azure)
Barge-in: cancel TTS on new speech

A voice agent that “feels slow” is often endpointing + LLM, not ASR model size. Measure each span with traces. Real-time AI is a system SLO, not a single vendor checkbox.

Interim vs Final vs Offline Truth

Interim text will flicker, rewrite numbers, and occasionally hallucinate endings. Show it in the caption bubble. Final text is what you send to CRM, tools, and LLMs. Offline re-ASR (Whisper large / batch hyperscaler / AssemblyAI) is the archival source of truth when compliance or search quality matters. Live and offline transcripts will diverge—store both with timestamps if you need to audit the agent.

Python: Streaming Recognize Pattern (Google-style, Portable Ideas)

The same state machine appears in Deepgram WS messages and Azure Recognizing/Recognized events. Swap the transport; keep the rules: small frames, handle silence, never block the send loop on LLM calls (use a queue).

from google.cloud import speech

client = speech.SpeechClient()
config = speech.RecognitionConfig(
    encoding=speech.RecognitionConfig.AudioEncoding.LINEAR16,
    sample_rate_hertz=16000,
    language_code="en-US",
    enable_automatic_punctuation=True,
    # model="telephony",  # confirm current streaming-capable model IDs
)
streaming_config = speech.StreamingRecognitionConfig(
    config=config,
    interim_results=True,
    single_utterance=False,
)

def audio_requests(frames):
    yield speech.StreamingRecognizeRequest(streaming_config=streaming_config)
    for chunk in frames:  # 20-100 ms PCM from mic / WebRTC tap
        yield speech.StreamingRecognizeRequest(audio_content=chunk)

def run_live(frames):
    committed = []
    for resp in client.streaming_recognize(audio_requests(frames)):
        if not resp.results:
            continue
        result = resp.results[0]
        text = result.alternatives[0].transcript
        if result.is_final:
            committed.append(text)
            yield ("final", text)     # CRM / agent / PII redact then log
        else:
            yield ("partial", text)   # UI only — do not persist

# for kind, text in run_live(mic_frames()): ...

Endpointing, Barge-in, and Overlap

Endpointing: silence duration that closes an utterance. Too short → cut-off users; too long → sluggish agents. Some vendors expose a millisecond parameter; still tune per locale and call type.
Barge-in: if the user speaks while TTS is playing, stop synthesis and treat new audio as a turn. Requires VAD on playback + streaming STT, not batch Whisper.
Overlap / two talkers: live diarization is weak; for captions, show text without fake names. For archives, re-run diarization offline.

PII on the Live Path

Partials in browser devtools, CDN logs, and LLM traces are still PII. Redact finals before they leave the speech service boundary when vendors support it; always redact before long-term storage and before Volume 15 agents. Prefer not to log raw partials in production. See privacy and compliance.

Vendor Cheat-Sheet (When to Pick What)

Job	Lean live STT	Lean offline STT	Lean TTS
Voice agent / IVR assist	Deepgram or hyperscaler streaming (estate-native)	Optional Whisper/batch re-decode	ElevenLabs / PlayHT / Azure neural
Live captions (events)	Streaming STT + interim UI	Not required	N/A
Meeting notes + intel	Optional live captions	AssemblyAI or hyperscaler analytics + diarize	Optional recap voice
Air-gap / PHI batch	Only if you have on-prem streaming containers	Whisper + pyannote	On-prem TTS or skip
AWS/GCP/Azure standardized	That cloud’s streaming STT	That cloud’s batch STT	That cloud’s TTS unless bake-off says otherwise

Stream when

Humans or agents must act mid-utterance.
Captions, barge-in, coaching overlays.
You can accept slightly higher live WER.

Batch when

Search, compliance, training data.
Best DER/WER/PII redaction.
Whisper-class multilingual archives.

Common Misconception

“If we chunk Whisper every 800 ms we have real-time transcription.” You have lagged batch with boundary artifacts. Real-time STT is a streaming decoder plus endpointing plus partial/final semantics. Chunked Whisper can back a low-stakes caption prototype; it will not meet voice-agent barge-in SLOs. Also: streaming WER from a vendor slide is not your production WER on packet-lossy mobile networks—test on the real transport.

Knowledge Check

Short Answer: What is the difference between interim and final transcripts? Answer: Interims are unstable UI hypotheses; finals are committed utterances after endpointing.
True/False: Streaming STT usually matches or beats offline WER on the same audio. Answer: False—offline/batch often wins accuracy.
Multiple Choice: Endpointing mainly controls: (a) embedding dimension, (b) when a live utterance is closed, (c) image CFG scale. Answer: (b).
Short Answer: Why is chunked Whisper not true streaming? Answer: Each chunk is still a batch decode with extra boundary errors and no native partial/final contract.
True/False: You should persist every interim partial to the CRM. Answer: False.
Multiple Choice: A common live+offline pattern is: (a) Deepgram live + Whisper archive, (b) PlayHT live ASR + FAISS STT, (c) DALL·E diarization. Answer: (a).
Short Answer: Name two places latency hides besides the ASR model. Answer: Any two of: jitter buffer, endpointing, network, LLM, TTS TTFB, barge-in handling.
True/False: Logging unredacted partials can leak PII even if finals are redacted later. Answer: True.
Multiple Choice: Best on-prem batch default in this module: (a) PlayHT STT, (b) Whisper open weights, (c) ElevenLabs only. Answer: (b).
Short Answer: Which Volume 16 module comes next after this lecture? Answer: 16.3 Image Generation (DALL·E first).

Key Takeaways

Real-time transcription is a streaming contract: partials, finals, endpointing, and a latency budget.
Batch still wins WER, DER, and often PII/intel—run both when the archive matters.
Pick live STT by estate + SLO (Deepgram vs hyperscalers); pick Whisper for on-prem/offline.
Do not log partials; redact finals; treat diarization as weaker live than offline.
Module 16.2 complete → 16.3 DALL·E.

Trainer’s Guide

Lab: Build the smallest live captioner (Deepgram WS or Azure continuous or Google streaming). Display partial vs final in two colors. Then re-transcribe the recording with Whisper and diff. Optionally add PlayHT/ElevenLabs TTS to measure barge-in.

Capstone discussion: For a hospital voice agent, choose live STT, offline STT, TTS, diarization, and PII controls. Students must justify on-prem vs cloud using this module’s catalog tables—no invented WER numbers.

Recap: Stream for interaction, batch for truth. Module 16.2 ends here—continue with DALL·E in 16.3.