← Master Index
Vol. 23 Module 23.1 Lecture

AI Meeting Summarizer

Capstone Projects

How This Lesson Fits the Module & Volume

The research assistant cited documents. A meeting summarizer cites utterances: who said what, what was decided, and who owns the next step. Vol. 16 speech-to-text, speaker diarization, and real-time transcription are the observe path. This lecture is the product: transcript → summary + decisions + action items, with consent and PII controls (Vol. 20 privacy).

Vol. 21 voice assistants and Document AI are siblings: one is live dialogue, one is files; meetings sit between. Next: multi-agent travel planner—agents with mock tools and a human booking gate.

Learning Objectives

By the end of this lesson, students should be able to:

  • Define the MVP: labeled transcript → summary, decisions, and action items with evidence spans.
  • Require recording/transcription consent before any STT or storage.
  • Keep speaker labels on commitments; refuse invented attendees or fake “we decided.”
  • Redact or flag PII; set a retention policy (Vol. 20 privacy).
  • Eval action-item precision/recall against a gold transcript—not ROUGE alone.
  • Place HITL before calendar writes or emailing minutes to the org.
Definition

An AI meeting summarizer turns a consented transcript (optionally produced by Vol. 16 STT + diarization) into a structured recap: narrative summary, decisions (agreements that change state), and action items (owner + task + optional due date) each tied to an evidence span (speaker + quote). It is not a silent recorder, not a surveillance product, and not allowed to invent people, votes, or deadlines that do not appear in the transcript.

Problem, MVP, and Stretch

MVP (ship this)Stretch (after eval is green)
InputPaste/upload transcript with speaker labelsAudio → Whisper/Deepgram/AssemblyAI (Vol. 16.2); live streaming
ConsentBoolean consent_recorded; reject if falsePer-jurisdiction notice copy; participant ack log
OutputsSummary + decisions[] + action_items[] + evidence spansTopic chapters, sentiment (use cautiously), multi-language
SpeakersHonor labels; unknown → “Speaker N”Diarization + roster match with HITL rename
PIIRegex/heuristic flags (emails, phones) + redact toggleNER redaction pipeline; tenant retention job
DownstreamDownload Markdown/JSON; human editsCalendar/task write only after HITL approve
Out of scopeRecording without notice; real customer audio in classAlways-on meeting spyware; auto-send to all-hands

Decision

  • “We will ship v1 on Friday”
  • Changes team state
  • Needs speakers who agreed
  • Must quote the transcript

Action item

  • “Alex will file the bug by Tuesday”
  • Has an owner and a verb
  • Optional due date only if spoken
  • Unassigned “someone should” ≠ an item

Not either

  • Brainstorming without close
  • Jokes, parking-lot chat
  • Model-invented owners
  • PII gossip to keep out of minutes

Structured minutes buy

  • Evalable fields (owner/task/due)
  • Evidence spans for HITL edit
  • Clearer privacy surface than a free-form essay

Prose-only recap costs

  • ROUGE can look high while owners are wrong
  • Invented decisions look authoritative in Slack
  • Hard to redact PII consistently

Architecture

LayerMVP choiceNotes
UIConsent checkbox + transcript paste + editable minutesShow speaker colors; highlight evidence on click
APIFastAPI POST /v1/meetings/summarizeReject consent_recorded=false
STT (optional)Skip in MVP; accept textStretch: Vol. 16 Whisper / vendor STT; store WER caveats
ExtractJSON schema via structured output (Vol. 13)Validate with Pydantic; no free-form “minutes blob” as the contract
PIIFlag + optional redact before persistVol. 20; never log raw audio in class demos
StorageSQLite: meeting_id, tenant, transcript hash, JSON recap, retention_untilDelete job is part of “done”
HITLHuman must confirm before share/calendarVol. 15 HITL; irreversible writes gated
Consent + transcript (or STT)
PII flag / redact
LLM → summary + decisions + actions (JSON)
Evidence-span check + HITL edit
Export or (stretch) calendar write

FastAPI Sketch: Transcript → Decisions & Actions

# meeting_summarizer.py — Vol. 23 capstone (educational) # Consent first. Cite utterances. Do not invent owners or decisions. import re from fastapi import FastAPI, HTTPException from pydantic import BaseModel, Field app = FastAPI(title="Vol23 Meeting Summarizer") PII_RE = re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+|\+?\d[\d\-\s()]{8,}\d") class Utterance(BaseModel): speaker: str = Field(min_length=1, max_length=80) ts_sec: float | None = None text: str = Field(min_length=1, max_length=4000) class MeetingIn(BaseModel): meeting_id: str tenant_id: str consent_recorded: bool utterances: list[Utterance] = Field(min_length=1, max_length=2000) class ActionItem(BaseModel): owner: str task: str due: str | None = None evidence_speaker: str evidence_span: str class Decision(BaseModel): text: str speakers: list[str] evidence_span: str class RecapOut(BaseModel): summary: str decisions: list[Decision] action_items: list[ActionItem] pii_flags: int status: str def transcript_blob(uts: list[Utterance]) -> str: return "\n".join(f"{u.speaker}: {u.text}" for u in uts) def evidence_in_transcript(span: str, blob: str) -> bool: s = " ".join(span.lower().split()) b = " ".join(blob.lower().split()) return len(s) >= 12 and s in b def llm_extract(blob: str) -> dict: # Replace with structured-output chat (Vol. 13). Return JSON dict. return {"summary": "Replace with model summary.", "decisions": [], "action_items": []} @app.post("/v1/meetings/summarize", response_model=RecapOut) def summarize(body: MeetingIn): if not body.consent_recorded: raise HTTPException(status_code=403, detail="Consent required before transcription or summarization.") blob = transcript_blob(body.utterances) pii_flags = len(PII_RE.findall(blob)) raw = llm_extract(blob) decisions = [Decision.model_validate(d) for d in raw.get("decisions", [])] items = [ActionItem.model_validate(a) for a in raw.get("action_items", [])] speakers = {u.speaker for u in body.utterances} for d in decisions: if not evidence_in_transcript(d.evidence_span, blob): raise HTTPException(status_code=422, detail="Decision evidence span not found in transcript.") for a in items: if a.owner not in speakers: raise HTTPException(status_code=422, detail=f"Invented owner: {a.owner}") if not evidence_in_transcript(a.evidence_span, blob): raise HTTPException(status_code=422, detail="Action evidence span not found in transcript.") return RecapOut( summary=str(raw.get("summary", "")), decisions=decisions, action_items=items, pii_flags=pii_flags, status="needs_hitl_share", )

Acceptance Criteria (“Done When…”)

#CriterionHow you prove it
1Consent gateRequest without consent → 403; no persist
2Schema recapJSON: summary, decisions, action_items with evidence spans
3No invented ownersOwner must be a speaker label present in the transcript
4Evidence spans resolveEach decision/action quote appears in the transcript (normalized)
5Decision vs action splitGold fixture: at least one of each classified correctly
6PII visibleEmail/phone flags counted; optional redacted export
7HITL before shareNo auto-email / auto-calendar in MVP
8RetentionStored recap has retention_until or delete path documented

Eval, HITL, and Safety

Do not score this product with summary ROUGE alone (Vol. 19 ROUGE is secondary). Primary: action-item precision / recall against gold (owner+task match), decision exactness, invented-entity rate = 0 on fixtures. STT error (WER) upstream will poison extraction—label ASR vs LLM blame separately when you add audio.

RiskControl
Recording without noticeConsent flag; product copy; refuse to run STT otherwise
PII in minutes / logsFlag, redact export, minimize prompts, retention job (Vol. 20)
Invented commitmentsEvidence span + owner ∈ speakers; HITL edit
Prompt injection via transcriptWrap utterances as untrusted data
Auto-writing calendarsHITL approve; mock calendar in class

Class rule: use synthetic transcripts you wrote. Do not upload real workplace recordings or student audio without institutional consent. Vendor STT (Vol. 22 / Vol. 16.2) is optional substrate—no fake WER leaderboards.

Related Lectures

LectureRole
STT / diarization / WhisperAudio → labeled text
Research assistantCitation discipline on docs vs utterances
Voice assistants / Document AIProduct siblings
Structured outputMinutes JSON contract
Privacy / HITLConsent, PII, share gate
Precision / recallAction-item eval
Travel plannerNext multi-agent capstone
Common Misconception

“If ROUGE is high, the minutes are usable.” Wrong owners still ship bad work. Second: diarization errors are the LLM’s fault—fix STT/labels first. Third: “someone should” is an action item (it is not, until there is an owner). Fourth: consent is implied because people joined a video call—your product still needs an explicit flag and notice. Fifth: dumping full transcripts into logs is fine if you summarized. Sixth: auto-creating calendar events without HITL is a feature rather than an incident waiting to happen.

Knowledge Check

  1. Short Answer: What three structured outputs should the MVP return besides a summary? Answer: Decisions, action items, and evidence spans (plus speaker labels / PII flags).
  2. True/False: The API may summarize without consent_recorded. Answer: False—reject (e.g. 403).
  3. Multiple Choice: Primary eval for action items: (a) precision/recall vs gold, (b) only ROUGE-L, (c) perplexity. Answer: (a).
  4. Short Answer: Why must an action-item owner appear in the speaker set? Answer: To block invented attendees/owners.
  5. True/False: “Someone should update the docs” is a valid MVP action item. Answer: False—no owner.
  6. Multiple Choice: Vol. 16 capability that labels who spoke: (a) diarization, (b) LoRA, (c) beam search only. Answer: (a).
  7. Short Answer: Name one Vol. 20 control required here. Answer: Consent, PII redaction/minimization, or retention (any one).
  8. True/False: Evidence spans must appear in the transcript. Answer: True.
  9. Multiple Choice: Calendar write in this capstone: (a) HITL or stretch only, (b) fire automatically, (c) billed as training FLOPs. Answer: (a).
  10. Short Answer: Which earlier Vol. 23 lecture used citation gates on documents? Answer: AI Research Assistant.

Key Takeaways

  • Meeting summarizers extract decisions and owned action items from consented transcripts—not essays without evidence.
  • Speaker labels, evidence spans, and invented-owner checks are acceptance tests.
  • Consent, PII, retention, and HITL share gates are product features (Vol. 16 + Vol. 20).
  • Eval action-item precision/recall; do not hide behind ROUGE.
  • Next: Multi-Agent Travel Planner.
Trainer’s Guide

Lab: Hand out three synthetic transcripts (standup, design review, 1:1). Gold JSON: 2 decisions + 3 action items each, plus one “someone should” trap and one invented-owner trap. Students implement consent + evidence + owner checks. Optional: run Whisper on a student-recorded reading of the synthetic script—never real workplace audio.

Discussion: When is a joke “we’ll just rewrite production tonight” a decision? Teach HITL: minutes are drafts until a human confirms.

Recap: Meeting summarizers productize Vol. 16 STT plus structured extraction—consent, speakers, decisions, action items, PII, and HITL. Continue to Multi-Agent Travel Planner.