Deepgram — Module 16.2

How This Lesson Fits the Module & Volume

Whisper covers multilingual batch ASR. Deepgram is the streaming-first cloud STT you reach for when a voice agent, live caption, or contact-center coach cannot wait for a file to finish. It sits between open Whisper and the hyperscalers: a specialist speech API with WebSockets, optional diarization, and redaction—not a general cloud estate.

Module 16.1’s real-time AI and speech-to-text lectures stated the capability. This lecture is the vendor decision: when Deepgram beats Whisper chunking, when AssemblyAI wins on audio intelligence, and when you stay inside Google/Azure/AWS. Full streaming mechanics land in real-time transcription.

Learning Objectives

By the end of this lesson, students should be able to:

Describe Deepgram as a cloud STT platform with pre-recorded and live (WebSocket) paths.
Contrast streaming vs batch on latency, WER posture, and cost SKUs.
Call the pre-recorded listen API and sketch a live socket with interim vs final text.
Enable diarization and PII redaction as product flags—and know their failure modes.
Compare Deepgram to Whisper, AssemblyAI, and hyperscalers on languages, on-prem, and pricing posture.
Decide when Deepgram is the default for voice agents vs when to keep audio in a hyperscaler VPC.

Definition

Deepgram is a speech-to-text (and related audio) cloud API. You send audio bytes—a file for batch, or a socket for live—and receive transcripts, optional word timings, speaker labels, and formatting. Model names (Nova family and hosted Whisper-class options) change; the engineering contract is low-latency listen + features as query flags.

Why Teams Reach for Deepgram

Voice products fail on time to first committed word, not on overnight WER. Deepgram’s product center of gravity is live recognition: interim hypotheses while the user is still talking, then a finalized utterance after endpointing. That is the opposite of Whisper’s file-in, transcript-out loop.

Batch still exists (pre-recorded listen). Use it for call archives, QA, and offline re-score. Many production designs run live Deepgram for the UX and optionally Whisper or a larger batch model overnight for a cleaner archive—two WER/latency budgets, one conversation.

Catalog Snapshot (Qualitative)

No invented WER. Model generations improve; measure on your telephony and headset audio.

Dimension	Deepgram	Whisper (self/API)	Hyperscaler STT
Accuracy posture	Strong on conversational/telephony when you pick a current Nova-class model; still eval locally	Strong multilingual batch; weaker live story	Broad domain models (medical, video, chirp-class); eval required
Streaming	First-class WebSocket; interim + final	Not native	First-class (gRPC / SDK / WS)
Languages	Multilingual list—verify locale before launch	Very broad	Typically the widest enterprise locale lists
Pricing posture	Per-minute; live vs pre-recorded often different SKUs	API per-minute or self-host GPUs	Per-15s / hour + commitment discounts
On-prem vs cloud	Cloud-first; enterprise dedicated / self-hosted options exist—confirm current offering	Open weights = true on-prem	Cloud + some container/on-prem SKUs
Diarization	`diarize` / utterances	External (pyannote)	Vendor diarization configs
PII	Redaction parameters on the request	DIY downstream	Redaction + cloud DLP story

Streaming vs Batch on Deepgram

Live (WebSocket)

Mic / PSTN / WebRTC tap
Interim text for UI
Endpointing / utterance close
Voice agents, captions

Pre-recorded

File or URL upload
Full-file context
Often cleaner punctuation
Archives, QA, training data

Feature flags

Smart formatting
Diarize / utterances
Keywords / keyterms
Redact PII

Pre-recorded Listen (REST sketch)

SDKs churn; REST query parameters are the stable teaching surface. Confirm model IDs and redaction enums in current Deepgram docs. Never hard-code API keys.

import os
import requests

API_KEY = os.environ["DEEPGRAM_API_KEY"]

with open("call.wav", "rb") as f:
    audio = f.read()

resp = requests.post(
    "https://api.deepgram.com/v1/listen",
    params={
        "model": "nova-2",          # confirm current Nova / Whisper hosted IDs
        "smart_format": "true",
        "diarize": "true",
        "utterances": "true",
        "redact": "pii",            # confirm supported redact values
    },
    headers={
        "Authorization": f"Token {API_KEY}",
        "Content-Type": "audio/wav",
    },
    data=audio,
    timeout=120,
)
resp.raise_for_status()
alt = resp.json()["results"]["channels"][0]["alternatives"][0]
print(alt["transcript"])
for word in alt.get("words", [])[:8]:
    print(word.get("speaker"), word.get("word"), word.get("start"))

Live Socket Pattern (Conceptual)

A live session is a bidirectional WebSocket: you stream PCM/Opus frames (often 20–100 ms), Deepgram streams JSON messages. Distinguish interim (unstable, for the caption bubble) from final (committed, for the agent / CRM). Closing silence is endpointing—too aggressive and you cut users off; too lax and the agent waits forever.

# Pseudocode / teaching sketch — use Deepgram's current WS URL + SDK in production
# wss://api.deepgram.com/v1/listen?model=nova-2&interim_results=true&punctuate=true

import json, os
import websocket  # example library; official SDK preferred

url = (
    "wss://api.deepgram.com/v1/listen"
    "?model=nova-2&interim_results=true&smart_format=true&endpointing=300"
)
ws = websocket.create_connection(
    url, header=[f"Authorization: Token {os.environ['DEEPGRAM_API_KEY']}"]
)

def on_msg(raw: str) -> None:
    msg = json.loads(raw)
    alt = msg.get("channel", {}).get("alternatives", [{}])[0]
    text = alt.get("transcript", "")
    if not text:
        return
    if msg.get("is_final"):
        print("FINAL:", text)      # agent turn / store
    else:
        print("partial:", text)    # UI only

# send binary audio frames from the mic in a loop, then ws.send(json.dumps({"type": "CloseStream"}))

Diarization, Keywords, and PII

diarize=true labels speaker turns (Speaker 0/1), not legal identities. Overlap, similar voices, and short backchannels still confuse systems—see speaker diarization. Keyword / keyterm boosting helps rare product names; it is not a license to skip a gold eval. PII redaction removes or masks entities in the transcript; it does not erase the original audio unless you design that retention policy yourself (privacy).

Pick Deepgram when

Voice agents, live captions, IVR assist.
You want STT + diarize + redact without a hyperscaler account.
Latency SLOs dominate over “already on AWS.”
You will still batch-retranscribe critical calls.

Pick something else when

Air-gap / open-weights only → Whisper.
Summaries, topics, LeMUR-style LLM over audio → AssemblyAI.
Mandatory GCP/Azure/AWS data boundary.
You primarily need TTS/voice cloning → ElevenLabs / PlayHT.

Common Misconception

“Streaming STT is just Whisper with a socket.” Live recognizers emit unstable partials, manage endpointing, and often trade a little accuracy for latency. Whisper-on-chunks is a different algorithm with different errors. If your demo uses Deepgram live and your compliance archive uses Whisper, you must eval both transcripts—they will not match word-for-word.

Knowledge Check

Short Answer: What is Deepgram’s primary product posture versus Whisper? Answer: Streaming-first cloud STT (live WebSocket) versus batch multilingual ASR.
True/False: Interim transcripts are always safe to persist as the system of record. Answer: False—only finals (or a later batch re-decode) should be stored.
Multiple Choice: Endpointing controls: (a) GPU dtype, (b) when a live utterance is considered finished, (c) S3 retention. Answer: (b).
Short Answer: Name two request flags you might enable for a call-center archive. Answer: Any two of: diarize, utterances, smart_format, redact/PII, keywords.
True/False: Deepgram diarization identifies a speaker’s legal name out of the box. Answer: False—it labels turns (e.g. Speaker 0), not enrolled identities.
Multiple Choice: Pricing posture is typically: (a) per-minute with live vs pre-recorded SKUs, (b) per GPU-hour only, (c) free on-prem forever. Answer: (a).
Short Answer: Why run live Deepgram and overnight Whisper? Answer: Live path hits latency SLOs; batch path often yields a cleaner archive transcript.
True/False: PII redaction on the transcript automatically deletes the source audio recording. Answer: False—audio retention is a separate policy.
Multiple Choice: Deepgram is a weak fit when: (a) you need sub-second captions, (b) you must air-gap open weights, (c) you have telephony audio. Answer: (b).
Short Answer: Which later lecture deepens interim vs final mechanics? Answer: Real-Time Transcription.

Key Takeaways

Deepgram is the streaming STT specialist: WebSocket partials + pre-recorded listen.
Compare live vs batch SKUs, languages, diarization, PII, and cloud vs dedicated deploy—not a single WER slide.
Keep finals (and optional offline re-ASR) as the system of record.
Diarize ≠ identity; redact ≠ audio deletion.
Next: AssemblyAI for transcription plus audio intelligence.

Trainer’s Guide

Lab: Hit pre-recorded listen on a two-speaker clip with and without diarize. Then (if keys allow) stream a mic and log interim vs final timestamps. Discuss which events you would send to an LLM agent.

Whiteboard: Latency budget for a voice bot: mic → network → ASR partial → LLM → TTS. Where Deepgram sits, and what happens if endpointing is 200 ms vs 1500 ms.

Recap: Deepgram is live-first STT with optional diarize and redact. Continue with AssemblyAI.