AssemblyAI — Module 16.2

How This Lesson Fits the Module & Volume

Whisper is ASR. Deepgram is live ASR. AssemblyAI is ASR plus an audio-intelligence layer: speaker labels, PII redaction, summaries, topics, sentiment, and LLM reasoning over the transcript (LeMUR-style products). You pick it when the transcript is an input to analytics—not only a caption.

That maps to Module 16.1 speech-to-text plus downstream agents from Volume 15: a meeting bot is an agent whose first tool is STT. Hyperscalers offer overlapping intelligence (call analytics); AssemblyAI is the independent specialist. Diarization detail is lecture 9.

Learning Objectives

By the end of this lesson, students should be able to:

Describe AssemblyAI as async (and streaming) STT with an audio-intelligence API surface.
Submit a transcription job with speaker labels and PII redaction, then poll or webhook for completion.
Contrast AssemblyAI vs Deepgram vs Whisper vs hyperscalers on streaming, languages, pricing, on-prem.
Explain why WER alone is insufficient when the product is summaries or CRM fields.
Place LeMUR / LLM-over-audio as a post-ASR stage with its own hallucinations and PII risks.
Choose AssemblyAI for meeting/call intelligence vs a streaming specialist for barge-in voice bots.

Definition

AssemblyAI is a cloud speech platform: you create a transcript job (URL or upload), optionally enable speaker labels, redaction, and language detection, then consume JSON plus higher-level features (summaries, sentiment, topics, LLM prompts over the transcript). Streaming APIs exist; the product’s historical center of gravity is high-quality async jobs + intelligence add-ons.

STT vs Audio Intelligence

A raw transcript answers “what words?” Call analytics answers “what happened?”—action items, compliance phrases, customer sentiment, who spoke how long. AssemblyAI packages many of those stages behind one API key so product teams do not wire pyannote + regex PII + an LLM chain on day one.

You still own evaluation. A perfect-looking summary can hide a bad WER on drug names or account numbers. Score ASR WER and downstream task accuracy (summary faithfulness, PII leak rate) separately.

Catalog Snapshot (Qualitative)

Dimension	AssemblyAI	Deepgram	Whisper	Hyperscalers
Accuracy posture	Strong async ASR; always eval on domain audio	Strong live + batch conversational	Strong multilingual batch	Domain models (medical, video, chirp)
Streaming	Available; async remains the “intelligence” home	Streaming-first	Not native	First-class streaming + batch
Languages	Growing list—check current docs per feature	Check locale list	Very broad	Typically widest enterprise lists
Pricing posture	Per-minute ASR + paid intelligence / LLM add-ons	Per-minute live vs pre-recorded	API minutes or GPUs	Tiered + commitments + extra APIs
On-prem vs cloud	Cloud-first	Cloud + some dedicated options	On-prem via open weights	Cloud + containers / VPC
Diarization	`speaker_labels` (and related)	`diarize`	External	Vendor diarization / conversation STT
PII / intel	First-class redact + summarization / LeMUR	Redact; intel more DIY or lighter	DIY	Call Analytics / DLP / Cognitive features

Job Lifecycle

1. Create

Upload or public URL
Config: speakers, PII, language
Webhook or poll

2. Transcribe

Async queue (seconds to minutes)
Utterances + timestamps
Status: queued / processing / error

3. Enrich

Summary, topics, sentiment
LLM prompt over transcript
Push to CRM / RAG index

Python: Transcribe with Speakers and PII Redaction

The official assemblyai SDK wraps create-and-wait. In production prefer webhooks over blocking wait, and never log unredacted transcripts. Feature names evolve—verify enums in current docs.

import os
import assemblyai as aai

aai.settings.api_key = os.environ["ASSEMBLYAI_API_KEY"]

config = aai.TranscriptionConfig(
    speaker_labels=True,
    language_detection=True,
    punctuate=True,
    format_text=True,
    redact_pii=True,
    redact_pii_policies=[
        "person_name",
        "phone_number",
        "email_address",
        "credit_card_number",
    ],
    redact_pii_redacted_audio=False,  # True only if you also need a scrubbed WAV
)

transcriber = aai.Transcriber(config=config)
transcript = transcriber.transcribe("https://example.com/meetings/standup.mp3")

if transcript.status == aai.TranscriptStatus.error:
    raise RuntimeError(transcript.error)

print(transcript.text)
for u in transcript.utterances or []:
    print(f"Speaker {u.speaker} [{u.start}-{u.end}]: {u.text}")

LLM-over-Audio (LeMUR-style)

Once you have a transcript ID, you can ask an LLM grounded on that transcript (summarize, extract action items, Q&A). This is not a substitute for ASR quality: the model cannot recover a drug name the recognizer dropped. Treat prompts as untrusted if users can inject speech that looks like instructions (prompt injection via audio). Keep PII out of prompt logs.

# Pattern only — method names track AssemblyAI's current LeMUR / LLM API
result = aai.Lemur().task(
    transcript_ids=[transcript.id],
    prompt="List action items as JSON: owner, due, text. Use only the transcript.",
    final_model=aai.LemurModel.claude3_5_sonnet,  # confirm available models
)
print(result.response)

Pick AssemblyAI when

Meetings, interviews, call analytics products.
You want speaker labels + PII + summary in one vendor.
Async latency (seconds–minutes) is acceptable.
An LLM must answer questions about the recording.

Pick something else when

Barge-in voice agent → Deepgram / hyperscaler streaming.
Air-gapped ASR → Whisper open weights.
Already standardized on AWS Transcribe Call Analytics / Azure.
You only need TTS → ElevenLabs / PlayHT.

Common Misconception

“If the summary looks fluent, the transcript is accurate.” LLMs repair grammar and invent plausible action items. Always sample-check WER on critical entities (names, dosages, amounts) and keep a human-in-the-loop on high-stakes outputs. Intelligence features multiply both ASR errors and LLM hallucinations.

Knowledge Check

Short Answer: How does AssemblyAI’s center of gravity differ from Deepgram’s? Answer: Async transcription + audio intelligence vs streaming-first live ASR.
True/False: Speaker labels are the same as verifying a person’s identity. Answer: False—they are diarization turns, not enrollment-based ID.
Multiple Choice: PII redaction on AssemblyAI typically acts on: (a) GPU kernels, (b) transcript entities (and optionally audio), (c) TLS certificates. Answer: (b).
Short Answer: Why score WER and summary quality separately? Answer: Fluent summaries can hide ASR entity errors and LLM invention.
True/False: AssemblyAI is the usual first pick for air-gapped on-prem ASR. Answer: False—it is cloud-first; use Whisper weights for air-gap.
Multiple Choice: Pricing posture is best described as: (a) ASR minutes plus intelligence/LLM add-ons, (b) one-time perpetual license only, (c) per-token image gen. Answer: (a).
Short Answer: Name a production alternative to blocking transcribe() wait. Answer: Webhooks (or async poll with backoff) when the job completes.
True/False: LeMUR-style LLM calls remove the need for a good transcript. Answer: False—the LLM is grounded on ASR output.
Multiple Choice: Best AssemblyAI use case: (a) sub-200 ms barge-in IVR, (b) meeting notes + action items, (c) image diffusion. Answer: (b).
Short Answer: Which hyperscaler lecture continues the “cloud estate + STT” theme? Answer: Google Speech-to-Text (next).

Key Takeaways

AssemblyAI = cloud ASR + intelligence (speakers, PII, summaries, LLM-over-audio).
Async jobs fit meetings and analytics; streaming specialists still win barge-in bots.
Eval WER and downstream task metrics; do not trust fluent summaries alone.
Cloud-first: if you need on-prem, Whisper (or hyperscaler containers) instead.
Next: Google Speech-to-Text.

Trainer’s Guide

Lab: Transcribe a two-speaker meeting with speaker_labels and PII redaction. Compare redacted vs raw text. Prompt the LLM layer for action items and mark invented items not in the audio.

Discussion: Where does AssemblyAI sit on a build-vs-buy whiteboard vs pyannote + Whisper + your own LLM chain (Volume 15)? Include privacy and log retention.

Recap: AssemblyAI is transcription plus audio intelligence. Continue with Google Speech-to-Text.