Whisper (OpenAI) — Module 16.2

How This Lesson Fits the Module & Volume

Module 16.1 defined the speech and audio modalities and the capability called speech-to-text. Module 16.2 is the engineering catalog: which product to run, how it streams, where audio lives, and how you measure error—not another recap of spectrograms.

Whisper is the right first vendor lecture because it is both an open-weight research model family and an OpenAI cloud API. Every later choice—Deepgram for streaming, hyperscalers for compliance estates, diarization, real-time transcription—is easier once you can contrast “self-host Whisper” against “buy a speech API.” Platform context also appears in Vol. 22 OpenAI.

Learning Objectives

By the end of this lesson, students should be able to:

Explain Whisper as an encoder–decoder ASR model (open weights) plus a hosted OpenAI Audio API.
Choose batch vs streaming posture: Whisper is batch-strong; low-latency live captions usually need another stack.
Run a transcription via the OpenAI API and via a local runtime such as faster-whisper.
Compare accuracy, languages, pricing posture, and on-prem vs cloud without inventing WER leaderboards.
State what Whisper does not give you natively: speaker diarization, PII redaction, and committed streaming partials.
Decide when Whisper is the right default vs Deepgram, AssemblyAI, or a hyperscaler.

Definition

Whisper is OpenAI’s family of automatic speech recognition (ASR) models: a multilingual encoder–decoder transformer trained on large weakly labeled audio/text pairs. “Whisper” in production may mean (1) open checkpoints you run yourself (tiny … large-v3), or (2) OpenAI’s hosted transcription endpoints (historically whisper-1, plus newer hosted transcribe models). Same research lineage; different ops, data-flow, and latency contracts.

What Problem Whisper Solves

Classic ASR stacks were language-specific, brittle to noise and accents, and expensive to train. Whisper popularized a single multilingual model that is surprisingly robust on “in the wild” audio—podcasts, lectures, phone calls—without a custom acoustic model per locale. That made good enough transcription a default feature instead of a research project.

It does not automatically solve live voice agents. The original architecture consumes relatively long audio windows; first-class interim hypotheses, barge-in, and sub-second committed words are the job of real-time transcription products. Treat Whisper as an excellent batch / near-batch engine unless you have explicitly chosen a streaming wrapper or a different vendor.

Catalog Snapshot (Qualitative)

Do not treat marketing WER as a purchase order. Run the same gold transcripts (your domain, your mics, your languages) through every candidate. The table below is an engineering posture comparison, not a benchmark.

Dimension	Whisper (open weights)	OpenAI Audio API	Streaming STT specialists
Accuracy posture	Strong multilingual ASR; large variants usually better, slower	Hosted Whisper-class / newer transcribe models; measure on your set	Often tuned for telephony + live partials; compare on your calls
Streaming	Not native; chunking hacks add latency/error	File/URL batch is the common path	WebSocket / SDK interim + final
Languages	Very broad multilingual coverage	Same family; check current model docs	Vendor list; some locales thinner
Pricing posture	GPU/CPU + engineering time	Per-minute (or successor unit); no GPU ops	Per-minute; stream vs pre-recorded SKUs
On-prem vs cloud	On-prem / air-gap possible	Cloud only; review data retention	Mostly cloud; some dedicated deploy
Diarization	Pair with pyannote / WhisperX	Not a speaker-ID product	Often a flag (lecture 9)
PII	You implement redaction downstream	Policy + your post-process	Often built-in redaction options

Open Weights vs Hosted API

Self-host

Data stays in your VPC.
Pick size: tiny → large-v3.
You own GPUs, queues, VAD.
Best for privacy-sensitive batch.

OpenAI API

No GPU fleet to babysit.
File upload / URL transcribe.
Vendor ToS + retention apply.
Best for product velocity.

Hybrid

API for prototypes.
faster-whisper in prod VPC.
Same eval harness both paths.
Swap if WER/latency regresses.

Word Error Rate (WER) Without Fake Numbers

WER = (substitutions + deletions + insertions) / reference word count. Lower is better. Whisper is widely reported as strong on public multilingual test sets, but your WER on medical dictation, factory noise, or code-switched speech can look nothing like a paper. Always:

Hold out a gold set with human transcripts (and timestamps if you care about alignment).
Normalize text the same way (punctuation, numerals, casing) before scoring.
Slice WER by language, SNR, speaker, and telephony vs studio—one average hides failure modes.
Never compare Vendor A’s published WER to Vendor B’s unless the audio and reference protocol match.

Batch Transcription — OpenAI API

SDK names and model IDs move; the pattern is stable: open a binary file, call transcriptions, choose a response shape. Use verbose_json when you need segment timestamps. Confirm current model IDs in OpenAI docs before shipping.

import os
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

with open("lecture.mp3", "rb") as audio:
    result = client.audio.transcriptions.create(
        model="whisper-1",                 # confirm current transcribe model IDs
        file=audio,
        response_format="verbose_json",
        timestamp_granularities=["segment"],
        # language="en",                   # omit to let the model detect
    )

print(result.text)
for seg in result.segments:
    print(f"{seg.start:6.2f}-{seg.end:6.2f}  {seg.text}")

Batch Transcription — Local faster-whisper

faster-whisper is a common production runtime (CTranslate2) for open checkpoints. Enable VAD to skip silence. Device and compute type are ops choices, not accuracy magic—validate WER after changing precision.

from faster_whisper import WhisperModel

model = WhisperModel("large-v3", device="cuda", compute_type="float16")
segments, info = model.transcribe(
    "lecture.wav",
    beam_size=5,
    vad_filter=True,
    word_timestamps=True,
)
print("detected language:", info.language, "prob:", round(info.language_probability, 3))
for s in segments:
    print(f"[{s.start:.2f}-{s.end:.2f}] {s.text.strip()}")

When to Pick Whisper

Pick Whisper when

Batch jobs: podcasts, lectures, archives, video captions.
You need on-prem or air-gapped ASR (open weights).
Language coverage matters more than sub-second partials.
You will add diarization / PII as separate stages.

Pick something else when

Live captions or voice agents need interim text (Deepgram, hyperscalers).
You want turn-key speaker labels + summaries (AssemblyAI).
Audio must stay inside an existing GCP/Azure/AWS compliance boundary.
You need native PII redaction SLAs, not a DIY regex.

Latency, Chunking, and “Fake Streaming”

Teams sometimes slice a live mic into 2–8 s chunks and run Whisper per chunk. That is near-real-time batch, not true streaming: you pay extra WER at chunk boundaries, duplicate words, and still wait for the whole window. If the product requirement is committed words under a few hundred milliseconds, design for a streaming ASR API (next lectures) and keep Whisper for offline re-transcription of the same session (higher accuracy archive).

Common Misconception

“Whisper includes speaker diarization and is a real-time STT engine.” Open Whisper transcribes what was said, not reliably who said it, and the canonical path is file-level (or long-chunk) inference. Speaker labels need a diarization model or a vendor flag. Live partials need a streaming recognizer. Conflating the three is how voice-agent prototypes miss latency and attribution SLAs.

PII, Retention, and Eval Hygiene

Speech often contains names, IDs, and health data. Self-hosting Whisper helps with privacy residency, but you still must redact transcripts before logs, search indexes, and LLM prompts. Hosted APIs require reading the current data-use and retention terms—do not assume training opt-out from memory. Build an eval set that includes PII-like tokens so redaction regressions are visible. See also compliance.

Knowledge Check

Short Answer: Name the two deployment meanings of “Whisper.” Answer: Open-weight models you self-host, and OpenAI’s hosted Audio transcription API.
True/False: Whisper’s original architecture is primarily a low-latency streaming recognizer with interim partials. Answer: False—it is batch / long-window ASR.
Multiple Choice: WER is: (a) words spoken per minute, (b) (S+D+I)/N on a reference transcript, (c) GPU utilization. Answer: (b).
Short Answer: Why must you measure WER on your own audio? Answer: Public or vendor numbers may not match your noise, accent, domain, or language mix.
True/False: Open Whisper natively returns stable speaker identities (Alice vs Bob). Answer: False—diarization is a separate stage or vendor feature.
Multiple Choice: Self-hosting Whisper is most justified when: (a) you want sub-100 ms partials, (b) audio must stay on-prem, (c) you need Azure conversation transcription. Answer: (b).
Short Answer: What does chunking a live mic into Whisper windows actually give you? Answer: Near-real-time batch, not true streaming partials—extra boundary errors and latency.
True/False: Pricing posture for the API is typically usage-based (e.g. per minute), while self-host is compute + ops. Answer: True.
Multiple Choice: A good next vendor when you need first-class WebSocket interim text is: (a) Deepgram, (b) PlayHT, (c) DALL·E. Answer: (a).
Short Answer: Where should PII redaction live if you self-host Whisper? Answer: In your pipeline after (or alongside) ASR—Whisper does not provide a redaction SLA.

Key Takeaways

Whisper is the multilingual batch ASR default: open weights and/or OpenAI API.
Compare vendors on streaming, languages, pricing posture, on-prem, diarization, and PII—not a single marketing WER.
Self-host for residency; use the API for speed to market; eval both with the same gold set.
Diarization and real-time partials are adjacent systems, not free Whisper features.
Next: Deepgram for streaming-first cloud STT.

Trainer’s Guide

Lab: Transcribe the same 3-minute clip with whisper-1 (or current API model) and local base vs large-v3. Score WER against a human transcript; discuss cost/latency vs error. Optionally add a noisy telephony version of the same script.

Discussion: Draw the data-flow for API vs VPC self-host. Where do audio bytes and transcripts sit? Who can read them? When would you still re-run Whisper offline after a live Deepgram session?

Recap: Whisper is the open/multilingual batch ASR pillar of Module 16.2. Continue with Deepgram when live streaming matters.