Real Time AI — Module 16.1

How This Lesson Fits the Module & Volume

Every 16.1 capability so far can run offline (batch STT, queued video gen, avatar render). Real-time AI is the capability of producing usable outputs within a human conversational budget—typically sub-second partials, barge-in, and streaming I/O. It is how Volume 15 agents feel alive on a phone or in a live avatar. Product streaming STT is in 16.2 real-time transcription; here we define latency, protocols, and design rules across modalities.

Learning Objectives

By the end of this lesson, students should be able to:

Define real-time vs streaming vs batch multimodal inference.
Budget latency for voice agents (STT partials, LLM TTFT, TTS first audio).
Explain VAD, barge-in, and endpointing.
Sketch a WebSocket/streaming Python client for STT.
Know which 16.1 tasks are inherently offline (long video gen) vs live.
Connect real-time speech to 16.2 streaming products without catalog lock-in.

Definition

Real-time AI is inference whose end-to-end latency and incrementality match an interactive human loop. Streaming means the model emits partial results before the full input is finished (e.g. STT hypotheses while still speaking). Batch/offline jobs may be faster than real-time in throughput but fail the interaction clock.

Latency Budget (Voice Agent)

Stage	What streams	Felt budget (order of mag.)
Capture + VAD	PCM frames	10–30 ms hop
Streaming STT	Partial transcripts	<200–400 ms to first words
Agent / LLM	Token TTFT	hundreds of ms
Streaming TTS	First audio chunk	<300 ms after text starts
Avatar lip sync (live)	Viseme frames	match audio clock

If any stage waits for the entire utterance or the entire LLM answer, the call feels broken. Volume 15 HITL still applies: you can stream and still pause for a human on high-risk tools.

Batch vs Stream vs Real-Time

Batch

Full file in, full result out
Whisper on a podcast
16.4 video jobs

Streaming

Partial outputs over time
Live captions
16.2 streaming STT

Full-duplex real-time

Listen and speak with barge-in
Voice agents, live avatars
Hardest ops + eval

Practical: Streaming STT Client

# Conceptual streaming STT (Deepgram-style websocket; vendor details in 16.2)
import json, time, websocket  # illustrative; use official SDK in production

def stream_mic_pcm(ws_url: str, pcm_iter, on_partial, on_final):
    ws = websocket.create_connection(ws_url)
    for chunk in pcm_iter:           # 20 ms 16 kHz mono frames
        ws.send(chunk, opcode=websocket.ABNF.OPCODE_BINARY)
        ws.settimeout(0.01)
        try:
            msg = json.loads(ws.recv())
        except Exception:
            continue
        if msg.get("is_final"):
            on_final(msg["transcript"])
        elif msg.get("transcript"):
            on_partial(msg["transcript"])
    ws.close()

# Agent: on_final -> Vol. 15 loop; on_partial -> UI only (do not tool-call on every partial).
# Barge-in: if VAD hears user during TTS, cancel TTS immediately.

What Can Be Real-Time?

Often real-time

STT, TTS, VAD, simple vision classify
Live captions, voice agents
3D avatar visemes

Usually offline

Long video generation (16.4)
Heavy OCR on 200-page scans
Hour-long video understanding (unless sampled live)

Common Misconception

“Real-time means the model is small.” Small models help, but the capability is system design: chunking, streaming protocols, speculative decoding, barge-in, and not blocking on tools. A huge LLM can still feel real-time if TTFT and TTS chunking are engineered; a tiny batch Whisper job on a 30-minute file is not real-time.

Knowledge Check

Short Answer: What distinguishes streaming from batch? Answer: Partial results before the full input/output is complete.
True/False: Video generation jobs in 16.4 are typically full-duplex real-time. Answer: False—usually queued/offline.
Multiple Choice: Barge-in means: (a) user can interrupt TTS, (b) OCR boxes, (c) CLIP temperature. Answer: (a).
Short Answer: Should agents tool-call on every STT partial? Answer: No—UI only; act on finals (or stable endpoints).
True/False: Module 16.2 includes a real-time transcription product lecture. Answer: True.
Multiple Choice: TTFT refers to: (a) time to first token, (b) time to first thumbnail, (c) teacher forcing. Answer: (a).
Short Answer: Name two stages in a voice latency budget. Answer: STT partials, LLM TTFT, TTS first audio, VAD (any two).
True/False: Real-time forbids Volume 15 HITL. Answer: False—you can stream and still pause on risky tools.
Multiple Choice: Live captions are: (a) streaming STT, (b) image inpaint, (c) k-means. Answer: (a).
Short Answer: Next lecture revisits which vision backbone? Answer: Vision Transformer (ViT).

Key Takeaways

Real-time AI = interactive latency + incremental I/O, not just small models.
Voice agents need streaming STT/TTS, VAD, and barge-in.
Batch still wins for long video gen, big OCR, offline analytics.
16.2 covers streaming STT products; this lecture is the capability.
Next: Vision Transformer (ViT).

Trainer’s Guide

Stopwatch lab: Measure time-to-first-caption and time-to-first-TTS-byte on a live mic. Change from batch Whisper to streaming STT and plot the difference.

Whiteboard: Full-duplex state machine: listening / thinking / speaking / barged-in. Tie back to Vol. 15 agent loop.

Recap: Real-time is a latency and streaming capability across modalities. Continue with Vision Transformer (ViT).