← Master Index
Vol. 16 Module 16.1 Lecture

Audio

Modalities & Capabilities

How This Lesson Fits the Module & Volume

Speech was spoken language. Audio is the wider waveform modality: music, environmental events, machine noise, silence, and speech mixed together. Agents that only run STT will miss a fire alarm, a product beep, or a song. This lecture places audio tagging, separation, and music tasks beside speech so students do not overload Whisper with non-linguistic sound.

16.2 remains the speech product catalog. General audio models are not that catalog—do not send a drum loop to Whisper and expect a useful “transcript.”

Learning Objectives

By the end of this lesson, students should be able to:

  • Distinguish audio (sound) from speech (spoken language).
  • List audio capabilities: classification, detection, separation, enhancement, music.
  • Describe spectrograms as the usual 2D view of waveforms (CNN/ViT friendly).
  • Route a clip to STT vs audio tagging vs both.
  • Call a simple audio-classification API/pipeline in Python.
  • Know when video soundtracks need audio + speech together.
Definition

Audio is any acoustic waveform. Audio understanding maps that waveform to event labels, embeddings, stems, or enhanced audio. Speech is a subset where the target is linguistic content. Music and sound-event tasks are audio even when no words exist.

Speech vs Audio Routing

If the user needs…Route toLecture / catalog
Words that were saidSTTSTT / 16.2
Spoken replyTTS / cloningTTS, cloning
“Was that a glass break?”Sound-event detectionThis lecture
Karaoke / stemsSource separationThis lecture
Genre / mood / similarityAudio embeddingsThis lecture (cf. Vol. 09 vectors)
Talking head + voiceSpeech + vision + videoAvatars

Capability Menu

Tag & detect

  • Clip-level labels (siren, laugh)
  • Frame-level event onsets
  • Scene / acoustic context

Separate & enhance

  • Speech vs noise vs music stems
  • Denoise / dereverb before STT
  • Boost WER by cleaning audio first

Represent

  • Spectrogram + CNN or ViT
  • Contrastive audio embeddings
  • Same cosine idea as CLIP/Vol. 09

Spectrograms: Images of Sound

A short-time Fourier transform turns a 1D waveform into a 2D time–frequency image. That is why Vol. 07 CNNs and Vol. 10 ViTs transfer so well to audio tagging: you are doing vision on spectrograms. Do not confuse that implementation trick with the vision modality (camera pixels).

# Audio tagging vs STT routing (Hugging Face + optional Whisper) import librosa import numpy as np from transformers import pipeline tagger = pipeline("audio-classification", model="MIT/ast-finetuned-audioset-10-10-0.4593") stt = pipeline("automatic-speech-recognition", model="openai/whisper-tiny") def route_clip(path: str, speech_hint: bool = True) -> dict: wav, sr = librosa.load(path, sr=16000, mono=True) tags = tagger({"array": wav, "sampling_rate": sr}) top = tags[0]["label"] out = {"top_audio_tag": top, "tags": tags[:5], "transcript": None} # If speech-like, also transcribe; never skip tagging on mixed scenes if speech_hint or any("speech" in t["label"].lower() for t in tags[:3]): out["transcript"] = stt(path)["text"] return out # Agent rule: alarms/music -> act on tags; words -> act on transcript.

Why Audio Tagging Still Matters in Voice Agents

Run audio models when

  • Safety / IoT / factory sounds
  • Music ID, podcast chapters by theme
  • Pre-STT denoise / VAD (voice activity)

STT alone fails when

  • No words (beep, crash, score)
  • Overlapping music drowning speech
  • You need “who/what made the sound”
Common Misconception

“Audio = speech-to-text.” STT is one audio downstream task specialized to language. A siren with no speech should never produce a hallucinated sentence. Route non-speech to tagging/separation; use 16.2 models only when linguistic content is the target.

Knowledge Check

  1. Short Answer: How is audio broader than speech? Answer: Audio includes any sound; speech is specifically spoken language.
  2. True/False: Whisper is the right first model for classifying a glass-break event. Answer: False—use sound-event / audio tagging.
  3. Multiple Choice: A spectrogram is: (a) a time–frequency image of sound, (b) a CLIP text prompt, (c) a tokenizer. Answer: (a).
  4. Short Answer: Why denoise before STT? Answer: Cleaner speech usually lowers WER.
  5. True/False: Module 16.2 is the full catalog of music source-separation products. Answer: False—16.2 is STT & voice models.
  6. Multiple Choice: Voice activity detection (VAD) is mainly: (a) finding speech vs silence/noise, (b) generating video, (c) OCR. Answer: (a).
  7. Short Answer: Name two non-speech audio capabilities. Answer: Event tagging, source separation, music embedding, enhancement (any two).
  8. True/False: CNNs can tag audio because spectrograms look like images. Answer: True (implementation transfer, not the vision modality).
  9. Multiple Choice: Podcast “what was said” vs “was there applause” needs: (a) STT only, (b) tagging only, (c) often both. Answer: (c).
  10. Short Answer: Next capability lecture after audio? Answer: OCR.

Key Takeaways

  • Audio = sound modality; speech is the language slice of audio.
  • Tag, separate, enhance, embed—do not force everything through STT.
  • Spectrograms let CNN/ViT skills transfer without calling it “vision.”
  • 16.2 catalogs voice STT/TTS products, not all audio AI.
  • Next: OCR—text living inside images.
Trainer’s Guide

Lab: Three clips—(1) clean speech, (2) siren + no words, (3) speech over music. Students must route each through tagger and/or STT and write the agent policy.

Link back: Draw Vol. 09 embedding cosine search but for audio clips (similar songs / similar events).

Recap: Audio covers non-linguistic sound; speech remains the language path. Continue with OCR.