Text-to-Speech — Module 16.1

How This Lesson Fits the Module & Volume

STT observes; text-to-speech (TTS) acts: text → waveform so agents, courses, and IVR can talk. Voice cloning is TTS plus a speaker identity. Streaming TTS is part of real-time AI. Vendor voices (ElevenLabs, PlayHT, Azure, Google, Amazon) live in Module 16.2—this lecture is the capability, SSML/prosody, and eval.

Learning Objectives

By the end of this lesson, students should be able to:

Define TTS vs cloning vs voice conversion.
Describe the classic stack: text frontend → acoustic model → vocoder.
Use SSML/prosody controls (rate, break, say-as) responsibly.
Stream first-audio vs render full files.
Call a TTS API in Python and log text+voice metadata.
Point to 16.2 for product voices without treating TTS as “one vendor.”

Definition

Text-to-speech synthesizes a spoken waveform from text (and optional markup for emphasis, pauses, and pronunciation). A vocoder turns acoustic features or latents into samples. Stock TTS uses a catalog voice; cloning binds a custom speaker embedding.

TTS in the Speech Family

Capability	Input	Output
STT	Audio	Text
TTS	Text (+ voice id)	Audio
Cloning	Enrollment + text	Audio in that identity
Lip sync	Audio + face	Video

Frontend → Acoustic → Vocoder

Text frontend

Normalize numbers, dates, URLs
Grapheme-to-phoneme / SSML
Language and locale

Prosody / acoustics

Duration, F0, emphasis
Emotion / style tokens
Sentence planning for agents

Vocoder / decoder

Waveform synthesis
Streaming chunk vocoders
Quality vs latency tradeoff

Practical TTS + Streaming Hint

from pathlib import Path
from openai import OpenAI

client = OpenAI()

def tts_file(text: str, voice: str = "alloy", out: str = "out.mp3") -> str:
    audio = client.audio.speech.create(
        model="gpt-4o-mini-tts",
        voice=voice,
        input=text,
    )
    Path(out).write_bytes(audio.read())
    return out

def tts_stream_chunks(text: str, voice: str = "alloy"):
    """First-byte latency matters more than full-file MOS for voice agents."""
    with client.audio.speech.with_streaming_response.create(
        model="gpt-4o-mini-tts",
        voice=voice,
        input=text,
    ) as resp:
        for chunk in resp.iter_bytes():
            yield chunk  # write to speaker / websocket immediately

# Frontend rule: expand "Dr." vs "Dr" before TTS; send SSML/say-as for IDs and currency.
# Do not TTS unredacted PII into logs or shared speakers.

Quality vs Interaction

Offline / narration

Highest MOS / naturalness
Full-paragraph context
Courses, audiobooks, ads

Conversational

Time-to-first-audio, barge-in
Sentence-level synthesis
Voice agents + live avatars

Common Misconception

“TTS quality is only MOS.” For agents, a slightly less pretty voice that streams in 200 ms beats a beautiful 3-second buffer. Also: TTS is not cloning—stock voices need no enrollment. Product names (ElevenLabs vs PlayHT) wait for 16.2.

Knowledge Check

Short Answer: What does TTS invert? Answer: STT—text→speech instead of speech→text.
True/False: Voice cloning is required for all TTS. Answer: False—stock voices have no enrollment.
Multiple Choice: A vocoder’s job is to: (a) turn acoustics/latents into a waveform, (b) OCR a page, (c) cluster patches. Answer: (a).
Short Answer: Why stream TTS in voice agents? Answer: Lower time-to-first-audio and allow barge-in.
True/False: ElevenLabs vs PlayHT feature charts belong in 16.1 not 16.2. Answer: False—16.2 is the voice catalog.
Multiple Choice: SSML is mainly for: (a) pronunciation/prosody markup, (b) CLIP prompts, (c) SQL. Answer: (a).
Short Answer: Name one TTS frontend job. Answer: Number/date normalization, G2P, locale, SSML (any one).
True/False: Lip sync typically consumes TTS/cloned audio. Answer: True.
Multiple Choice: MOS-first narration vs agent TTS: (a) offline quality vs streaming latency, (b) OCR vs STT, (c) Vol. 05 vs Vol. 06. Answer: (a).
Short Answer: Next lecture? Answer: Image captioning.

Key Takeaways

TTS is text→speech; cloning adds identity; streaming adds latency rules.
Frontend + acoustics + vocoder; SSML for control.
Agents optimize first audio + barge-in, not only MOS.
Compare vendors in 16.2.
Next: Image captioning.

Trainer’s Guide

Lab: Same paragraph via file TTS vs streamed chunks. Measure TTFB. Then mis-normalize a phone number and hear why the frontend matters.

Chain: STT → agent → TTS round-trip from the opening speech lecture; now students can name each box precisely.

Recap: TTS speaks agent text; 16.2 lists voices. Continue with Image captioning.