Speaker Diarization — Module 16.2

How This Lesson Fits the Module & Volume

Every vendor flag you saw—Deepgram diarize, AssemblyAI speaker_labels, Google/Azure/AWS speaker configs, ElevenLabs diarize—is the same problem with different wrappers: who spoke when. This lecture is the cross-cutting engineering model so you can eval those flags instead of treating them as magic.

It connects speech / STT to downstream meeting notes and call analytics. Open-source reference: pyannote (and WhisperX-style ASR+align+diarize pipelines). Real-time constraints continue in lecture 10. Identity misuse ties to privacy and voice cloning (enrollment ≠ diarization).

Learning Objectives

By the end of this lesson, students should be able to:

Define diarization vs ASR vs speaker recognition (identification/verification).
Describe the classic pipeline: VAD → segmentation → embeddings → clustering / neural assignment.
Explain DER (diarization error rate) qualitatively—without fake leaderboard numbers.
Compare vendor diarization flags vs self-host pyannote on streaming, languages, pricing, on-prem.
Handle overlap, unknown speaker count, and PII when labeling turns.
Choose batch diarization for archives vs constrained live “speaker change” UX.

Definition

Speaker diarization answers “who spoke when?” It partitions an audio timeline into labeled turns (Speaker A/B or 0/1) without necessarily knowing legal names. It is not transcription (that is ASR) and not “is this Alice?” (that is speaker recognition / verification, which requires enrollment samples and a different threat model).

Three Problems People Conflate

ASR

What words?
Metric: WER
Whisper / STT APIs

Diarization

Which turn / speaker label?
Metric: DER
pyannote / vendor flags

Recognition

Is this enrolled identity X?
FAR / FRR style metrics
Azure speaker rec., custom

Pipeline (Engineering, Not Vendor Internals)

Most systems, open or cloud, follow the same stages. Vendors hide them behind one boolean; you still debug the stages when DER explodes.

1. VAD

Find speech vs silence/noise.

2. Segment

Cut candidate turns; overlap is hard.

3. Embed

Speaker embedding per chunk (x-vector / ECAPA-style).

4. Assign

Cluster or neural diarize → labels + times.

ASR can run before, after, or jointly. WhisperX-style pipelines transcribe with Whisper, force-align words, then attach pyannote labels to words. Vendor APIs often return word-level speaker fields already merged.

DER Without Fake Benchmarks

Diarization Error Rate combines missed speech, false alarm speech, and speaker confusion, usually scored with a collar (forgiveness window) around boundaries and sometimes ignoring overlap. Lower is better. Published DER on conversational telephone speech will not predict your all-hands recording with a single table mic. Always:

Annotate a small gold set with turn boundaries and speaker ids (even anonymous A/B/C).
Report DER and WER; a system can nail words and swap speakers.
Slice by overlap ratio, speaker count, and channel (headset vs far-field).
Never compare Vendor A’s marketing DER to pyannote’s paper DER on a different corpus.

Catalog Snapshot (Qualitative)

Dimension	Vendor flags (DG / AAI / GCP / Azure / AWS)	pyannote / WhisperX (self-host)	Live-only “speaker change”
Accuracy posture	Good on clean 2-speaker calls; degrades with overlap, similar voices, unknown N	Often best control for batch; you tune clustering / max speakers	Coarse; not archive-grade DER
Streaming	Some expose live speaker labels; quality usually worse than offline	Mostly offline / chunked	Designed for live UX
Languages	Tied to the STT locale; embeddings more language-agnostic than ASR but not magic	Embedding models are reasonably language-robust; still eval	Locale follows STT
Pricing posture	Often included or a small adder on STT minutes	GPU + engineering time	Part of streaming STT bill
On-prem vs cloud	Follows the STT vendor	True on-prem possible (GPU)	Follows streaming vendor
PII	Labels are pseudonymous until you map them; mapping creates identity data	Same—your mapping table is PII	Don’t display real names without enrollment + policy

Python: pyannote Offline Diarization

You need a Hugging Face token accepted for the pyannote model license. This does not transcribe; merge with Whisper/faster-whisper timestamps yourself or use WhisperX.

import os
from pyannote.audio import Pipeline

pipeline = Pipeline.from_pretrained(
    "pyannote/speaker-diarization-3.1",
    use_auth_token=os.environ["HF_TOKEN"],
)
# pipeline.to(torch.device("cuda"))  # if GPU available

diarization = pipeline("standup.wav")  # optional: {"min_speakers": 2, "max_speakers": 4}

for turn, _, speaker in diarization.itertracks(yield_label=True):
    print(f"{speaker} {turn.start:.2f} {turn.end:.2f}")

# Merge with ASR: assign each word timestamp to the overlapping turn label.
# Overlap regions: either drop, mark BOTH, or keep the longer turn — product choice.

Python: Vendor Path (AssemblyAI-style Merge)

Same idea as Deepgram utterances or AWS speaker labels: iterate turns already fused with text. Always keep original offsets for eval.

# After AssemblyAI / Deepgram / Transcribe job completes:
for u in transcript.utterances:  # or words[] with speaker field
    print(f"{u.speaker}\t{u.start}\t{u.end}\t{u.text}")

# Map pseudonymous labels to names ONLY if:
# - a human assigned them, or
# - an enrolled speaker-id model verified them,
# and the mapping is stored under your PII policy.

Failure Modes You Must Plan For

Overlap: two people talking; many systems emit one label or drop a turn.
Unknown K: min/max speaker hints help but wrong bounds cause merges or splits.
Similar voices / same headset EQ: embeddings collide.
Far-field single mic: room acoustics dominate identity cues.
Short backchannels: “yeah” attributed to the wrong speaker, poisoning LLM summaries.
Live vs batch: online diarization cannot look far into the future; offline DER is usually better.

Use vendor diarization when

2-party telephony, clean channels.
You already pay for that STT.
Good-enough labels for captions/CRM.
No GPU appetite for pyannote.

Self-host pyannote / WhisperX when

On-prem / sensitive audio.
You need to tune clustering and overlap policy.
Batch archives where DER is contractual.
You must combine Whisper WER with diarize.

Common Misconception

“Diarization tells us it was Dr. Patel speaking, so we can auto-file under her patient chart.” Diarization only says a speaker differed from another. Attaching a legal identity requires enrollment, a verification model, and a privacy review. Auto-naming from a calendar invite is a heuristic, not evidence—and it can leak PII into every downstream LLM prompt.

Knowledge Check

Short Answer: What question does diarization answer? Answer: Who spoke when (turn labels on a timeline), not necessarily legal identity.
True/False: WER already includes speaker-swap errors. Answer: False—WER ignores who said the words; use DER (and task metrics) too.
Multiple Choice: Speaker recognition differs because it: (a) only removes silence, (b) matches audio to an enrolled identity, (c) is identical to WER. Answer: (b).
Short Answer: Name the four classic pipeline stages. Answer: VAD, segmentation, embeddings, clustering/assignment.
True/False: Offline diarization usually has an easier job than live diarization. Answer: True—it can use future context.
Multiple Choice: Mapping Speaker 0 → a real name creates: (a) no new data, (b) identity/PII data, (c) a TTS voice. Answer: (b).
Short Answer: Why do overlapping speakers break naive systems? Answer: Embeddings mix or one turn is dropped/mis-assigned, inflating confusion and misses.
True/False: pyannote can be run on-prem with open (licensed) checkpoints. Answer: True (license/token permitting).
Multiple Choice: Vendor diarize=true is closest to: (a) a full speaker-ID courtroom system, (b) a packaged diarization stage on STT, (c) Whisper pretraining. Answer: (b).
Short Answer: Which lecture covers live partial transcripts next? Answer: Real-Time Transcription.

Key Takeaways

Diarization ≠ ASR ≠ speaker identification; measure DER and WER separately.
Pipeline: VAD → segments → embeddings → labels; vendors wrap this as a flag.
Overlap, unknown speaker count, and far-field mics are the usual failure modes.
Self-host pyannote/WhisperX for control and on-prem; use vendor flags for simple calls.
Next: Real-Time Transcription.

Trainer’s Guide

Lab: Run pyannote on a two-speaker clip, then the same clip through one cloud API with diarize on. Draw a timeline comparison. Introduce a deliberate overlap section and discuss policy (drop vs double-label).

Ethics: Should a meeting bot auto-attach names from the invite list? Debate false attribution vs UX, citing privacy.

Recap: Diarization labels turns, not legal identities. Continue with Real-Time Transcription.