ElevenLabs (STT/TTS) — Module 16.2

How This Lesson Fits the Module & Volume

Lectures 1–6 optimized recognition. ElevenLabs flips the catalog: the company is famous for generative TTS and voice cloning, and now also ships STT (Scribe-class). You study it here because real products are loops—hear the user (STT) → reason → speak (TTS)—and ElevenLabs is often the TTS half even when Deepgram or Whisper is the STT half.

Compare with PlayHT (also TTS-first) and Azure neural TTS. Real-time duplex design is lecture 10 plus 16.1 real-time AI. Consent and misuse belong with responsible AI.

Learning Objectives

By the end of this lesson, students should be able to:

Describe ElevenLabs as a generative voice platform (TTS, cloning, agents) with added STT.
Call TTS and STT from Python using current SDK/REST patterns.
Contrast ElevenLabs STT vs Whisper/Deepgram/hyperscalers on streaming, languages, pricing, on-prem.
Design a hybrid: specialist STT + ElevenLabs TTS when that wins latency or voice quality.
State consent, watermarking, and PII risks for cloned voices and stored audio.
Know when Azure/Google TTS is enough vs when a generative vendor is justified.

Definition

ElevenLabs is a cloud generative-voice company. Its core product is neural text-to-speech with instant/professional voice cloning, low-latency streaming TTS, and conversational-agent tooling. Speech-to-text (Scribe and successors) transcribes audio with optional event tags and diarization flags. Treat TTS as the reason most teams adopt ElevenLabs; treat STT as a convenience unless a bake-off shows it matches your ASR SLO.

TTS-First, STT-Second — Why That Matters

Buying STT from a TTS vendor can simplify a demo (one API key, one voice agent). Production voice agents often split vendors: Deepgram or a hyperscaler for committed words, ElevenLabs for spoken replies. That is normal. One vendor for both only wins if latency, WER, languages, and DPA all clear the same bar.

Catalog Snapshot (Qualitative)

Dimension	ElevenLabs	PlayHT	Azure / Google TTS+STT	Whisper / Deepgram
Accuracy posture (STT)	Usable Scribe-class ASR—eval vs specialists on your audio	TTS-first; do not assume STT leadership	Mature hyperscaler STT	ASR specialists / OSS
Streaming	Excellent streaming TTS; STT more file/near-real-time than Deepgram-class WS	Streaming TTS first	First-class STT streaming + TTS	Deepgram live STT; Whisper batch
Languages	Strong multilingual TTS; STT coverage—check current list	Multilingual TTS	Very broad STT+TTS catalogs	Whisper very broad STT
Pricing posture	Typically character-based TTS + minute-based STT / plans—confirm current sheet	Character / plan TTS	Metered STT hours + TTS chars	STT minutes or GPU
On-prem vs cloud	Cloud	Cloud	Cloud + some containers	Whisper on-prem possible
Diarization	STT diarize flag where offered	Not the reason to pick	Vendor STT diarization	Deepgram / pyannote
Voice / PII	Cloning + consent; audio retention; optional watermarking features—read current policy	Same cloning ethics	Enterprise DPA + neural voices	PII via redact or DIY

Voice Loop Building Blocks

TTS / clone

Brand or personal voice
SSML-like controls / style
Streaming audio out

STT (Scribe)

File → text (+ timestamps)
Optional diarize / events
Eval vs Whisper/Deepgram

Agents

STT → LLM → TTS
Barge-in needs fast STT
Often hybrid vendors

Python: TTS Convert

Voice IDs and model IDs change. Keep them in config. Stream when the user is waiting; write files for batch narration (courses, IVR prompts).

import os
from elevenlabs import ElevenLabs, save

client = ElevenLabs(api_key=os.environ["ELEVEN_API_KEY"])

audio = client.text_to_speech.convert(
    voice_id="JBFqnCBsd6RMkjVDRZzb",          # replace with your consented voice
    model_id="eleven_multilingual_v2",       # confirm current TTS model IDs
    text="Your appointment is Tuesday at 3 PM. Reply stop to cancel reminders.",
    output_format="mp3_44100_128",
)
save(audio, "reminder.mp3")

Python: STT (Scribe-class)

Confirm model id (scribe_v1 or successor) and diarization parameters in live docs. If WER on telephony is worse than Deepgram/Whisper on your gold set, keep ElevenLabs for TTS only.

import os
from elevenlabs import ElevenLabs

client = ElevenLabs(api_key=os.environ["ELEVEN_API_KEY"])

with open("call.mp3", "rb") as f:
    result = client.speech_to_text.convert(
        file=f,
        model_id="scribe_v1",     # confirm current STT model id
        tag_audio_events=True,
        diarize=True,
    )

print(getattr(result, "text", result))
# If the payload includes words/speakers, iterate them for a timeline — schema evolves.

Hybrid Reference Architecture

A common production pattern: Deepgram or Azure streaming STT → LLM (Volume 15 agent) → ElevenLabs streaming TTS. Whisper may re-transcribe the recording for CRM. Do not run three STTs on the live path unless you are explicitly A/B testing—latency stacks. Log voice IDs, model IDs, and consent receipts next to every synthesized clip.

Pick ElevenLabs when

Voice quality / cloning is the differentiator.
You need low-latency spoken replies.
Multilingual narration or dubbing workflows.
STT bake-off is “good enough” for your domain.

Pick something else when

ASR accuracy/PII/on-prem dominate → Whisper/hyperscaler/Deepgram.
Microsoft TTS+STT compliance bundle is mandatory.
You only need cheap IVR prompts → cloud neural TTS may suffice.
No lawful consent to clone a voice.

Common Misconception

“If they clone voices this well, their STT must also be state of the art for live agents.” TTS and ASR are different models, datasets, and latency stacks. A cinematic clone does not imply Deepgram-class interim partials or Whisper-class multilingual WER. Bake off STT and TTS independently—and never clone a voice without documented consent.

Knowledge Check

Short Answer: What is ElevenLabs primarily known for in this catalog? Answer: Generative TTS and voice cloning, with STT as an additional capability.
True/False: You must use ElevenLabs STT if you use ElevenLabs TTS. Answer: False—hybrid STT+TTS vendors are common and often better.
Multiple Choice: Pricing posture is typically: (a) TTS characters + STT minutes/plans, (b) only S3 PUT fees, (c) per GPU Whisper license. Answer: (a).
Short Answer: Why split Deepgram STT and ElevenLabs TTS? Answer: Optimize live recognition latency/WER separately from spoken voice quality.
True/False: ElevenLabs is the default air-gapped on-prem ASR. Answer: False—it is cloud; use Whisper weights for air-gap ASR.
Multiple Choice: Voice cloning without consent is: (a) a standard eval trick, (b) an ethics/legal failure, (c) required for WER. Answer: (b).
Short Answer: Name one STT feature flag you might see on Scribe-class APIs. Answer: Diarize and/or tag audio events (or timestamps)—confirm current API.
True/False: Streaming TTS quality implies streaming STT parity with Deepgram. Answer: False.
Multiple Choice: Best ElevenLabs-first use case: (a) GCS long-running medical ASR only, (b) expressive agent voice / narration, (c) FAISS indexing. Answer: (b).
Short Answer: Which TTS-first vendor lecture follows? Answer: PlayHT.

Key Takeaways

ElevenLabs is a generative voice platform; STT exists but must earn its place in a bake-off.
Hybrid architectures (specialist STT + ElevenLabs TTS) are a feature, not a failure.
Cloud-only; character/minute pricing; cloning requires consent and retention hygiene.
Eval TTS (listening tests) and STT (WER/PII) as separate scorecards.
Next: PlayHT.

Trainer’s Guide

Lab: Synthesize the same script with Azure TTS and ElevenLabs. Blind A/B with the class. Then transcribe a noisy call with ElevenLabs STT vs Whisper; compare proper nouns. Discuss hybrid design.

Ethics: Walk a consent checklist for cloning a colleague’s voice for an internal demo. Tie to responsible AI and voice cloning.

Recap: ElevenLabs leads on generative TTS; STT is optional and must be evaluated. Continue with PlayHT.