The research assistant cited documents. A meeting summarizer cites utterances: who said what, what was decided, and who owns the next step. Vol. 16 speech-to-text, speaker diarization, and real-time transcription are the observe path. This lecture is the product: transcript → summary + decisions + action items, with consent and PII controls (Vol. 20 privacy).
Vol. 21 voice assistants and Document AI are siblings: one is live dialogue, one is files; meetings sit between. Next: multi-agent travel planner—agents with mock tools and a human booking gate.
Learning Objectives
By the end of this lesson, students should be able to:
- Define the MVP: labeled transcript → summary, decisions, and action items with evidence spans.
- Require recording/transcription consent before any STT or storage.
- Keep speaker labels on commitments; refuse invented attendees or fake “we decided.”
- Redact or flag PII; set a retention policy (Vol. 20 privacy).
- Eval action-item precision/recall against a gold transcript—not ROUGE alone.
- Place HITL before calendar writes or emailing minutes to the org.
An AI meeting summarizer turns a consented transcript (optionally produced by Vol. 16 STT + diarization) into a structured recap: narrative summary, decisions (agreements that change state), and action items (owner + task + optional due date) each tied to an evidence span (speaker + quote). It is not a silent recorder, not a surveillance product, and not allowed to invent people, votes, or deadlines that do not appear in the transcript.
Problem, MVP, and Stretch
| MVP (ship this) | Stretch (after eval is green) | |
|---|---|---|
| Input | Paste/upload transcript with speaker labels | Audio → Whisper/Deepgram/AssemblyAI (Vol. 16.2); live streaming |
| Consent | Boolean consent_recorded; reject if false | Per-jurisdiction notice copy; participant ack log |
| Outputs | Summary + decisions[] + action_items[] + evidence spans | Topic chapters, sentiment (use cautiously), multi-language |
| Speakers | Honor labels; unknown → “Speaker N” | Diarization + roster match with HITL rename |
| PII | Regex/heuristic flags (emails, phones) + redact toggle | NER redaction pipeline; tenant retention job |
| Downstream | Download Markdown/JSON; human edits | Calendar/task write only after HITL approve |
| Out of scope | Recording without notice; real customer audio in class | Always-on meeting spyware; auto-send to all-hands |
Decision
- “We will ship v1 on Friday”
- Changes team state
- Needs speakers who agreed
- Must quote the transcript
Action item
- “Alex will file the bug by Tuesday”
- Has an owner and a verb
- Optional due date only if spoken
- Unassigned “someone should” ≠ an item
Not either
- Brainstorming without close
- Jokes, parking-lot chat
- Model-invented owners
- PII gossip to keep out of minutes
Structured minutes buy
- Evalable fields (owner/task/due)
- Evidence spans for HITL edit
- Clearer privacy surface than a free-form essay
Prose-only recap costs
- ROUGE can look high while owners are wrong
- Invented decisions look authoritative in Slack
- Hard to redact PII consistently
Architecture
| Layer | MVP choice | Notes |
|---|---|---|
| UI | Consent checkbox + transcript paste + editable minutes | Show speaker colors; highlight evidence on click |
| API | FastAPI POST /v1/meetings/summarize | Reject consent_recorded=false |
| STT (optional) | Skip in MVP; accept text | Stretch: Vol. 16 Whisper / vendor STT; store WER caveats |
| Extract | JSON schema via structured output (Vol. 13) | Validate with Pydantic; no free-form “minutes blob” as the contract |
| PII | Flag + optional redact before persist | Vol. 20; never log raw audio in class demos |
| Storage | SQLite: meeting_id, tenant, transcript hash, JSON recap, retention_until | Delete job is part of “done” |
| HITL | Human must confirm before share/calendar | Vol. 15 HITL; irreversible writes gated |
FastAPI Sketch: Transcript → Decisions & Actions
Acceptance Criteria (“Done When…”)
| # | Criterion | How you prove it |
|---|---|---|
| 1 | Consent gate | Request without consent → 403; no persist |
| 2 | Schema recap | JSON: summary, decisions, action_items with evidence spans |
| 3 | No invented owners | Owner must be a speaker label present in the transcript |
| 4 | Evidence spans resolve | Each decision/action quote appears in the transcript (normalized) |
| 5 | Decision vs action split | Gold fixture: at least one of each classified correctly |
| 6 | PII visible | Email/phone flags counted; optional redacted export |
| 7 | HITL before share | No auto-email / auto-calendar in MVP |
| 8 | Retention | Stored recap has retention_until or delete path documented |
Eval, HITL, and Safety
Do not score this product with summary ROUGE alone (Vol. 19 ROUGE is secondary). Primary: action-item precision / recall against gold (owner+task match), decision exactness, invented-entity rate = 0 on fixtures. STT error (WER) upstream will poison extraction—label ASR vs LLM blame separately when you add audio.
| Risk | Control |
|---|---|
| Recording without notice | Consent flag; product copy; refuse to run STT otherwise |
| PII in minutes / logs | Flag, redact export, minimize prompts, retention job (Vol. 20) |
| Invented commitments | Evidence span + owner ∈ speakers; HITL edit |
| Prompt injection via transcript | Wrap utterances as untrusted data |
| Auto-writing calendars | HITL approve; mock calendar in class |
Class rule: use synthetic transcripts you wrote. Do not upload real workplace recordings or student audio without institutional consent. Vendor STT (Vol. 22 / Vol. 16.2) is optional substrate—no fake WER leaderboards.
Related Lectures
| Lecture | Role |
|---|---|
| STT / diarization / Whisper | Audio → labeled text |
| Research assistant | Citation discipline on docs vs utterances |
| Voice assistants / Document AI | Product siblings |
| Structured output | Minutes JSON contract |
| Privacy / HITL | Consent, PII, share gate |
| Precision / recall | Action-item eval |
| Travel planner | Next multi-agent capstone |
“If ROUGE is high, the minutes are usable.” Wrong owners still ship bad work. Second: diarization errors are the LLM’s fault—fix STT/labels first. Third: “someone should” is an action item (it is not, until there is an owner). Fourth: consent is implied because people joined a video call—your product still needs an explicit flag and notice. Fifth: dumping full transcripts into logs is fine if you summarized. Sixth: auto-creating calendar events without HITL is a feature rather than an incident waiting to happen.
Knowledge Check
- Short Answer: What three structured outputs should the MVP return besides a summary? Answer: Decisions, action items, and evidence spans (plus speaker labels / PII flags).
- True/False: The API may summarize without
consent_recorded. Answer: False—reject (e.g. 403). - Multiple Choice: Primary eval for action items: (a) precision/recall vs gold, (b) only ROUGE-L, (c) perplexity. Answer: (a).
- Short Answer: Why must an action-item owner appear in the speaker set? Answer: To block invented attendees/owners.
- True/False: “Someone should update the docs” is a valid MVP action item. Answer: False—no owner.
- Multiple Choice: Vol. 16 capability that labels who spoke: (a) diarization, (b) LoRA, (c) beam search only. Answer: (a).
- Short Answer: Name one Vol. 20 control required here. Answer: Consent, PII redaction/minimization, or retention (any one).
- True/False: Evidence spans must appear in the transcript. Answer: True.
- Multiple Choice: Calendar write in this capstone: (a) HITL or stretch only, (b) fire automatically, (c) billed as training FLOPs. Answer: (a).
- Short Answer: Which earlier Vol. 23 lecture used citation gates on documents? Answer: AI Research Assistant.
Key Takeaways
- Meeting summarizers extract decisions and owned action items from consented transcripts—not essays without evidence.
- Speaker labels, evidence spans, and invented-owner checks are acceptance tests.
- Consent, PII, retention, and HITL share gates are product features (Vol. 16 + Vol. 20).
- Eval action-item precision/recall; do not hide behind ROUGE.
- Next: Multi-Agent Travel Planner.
Lab: Hand out three synthetic transcripts (standup, design review, 1:1). Gold JSON: 2 decisions + 3 action items each, plus one “someone should” trap and one invented-owner trap. Students implement consent + evidence + owner checks. Optional: run Whisper on a student-recorded reading of the synthetic script—never real workplace audio.
Discussion: When is a joke “we’ll just rewrite production tonight” a decision? Teach HITL: minutes are drafts until a human confirms.
Recap: Meeting summarizers productize Vol. 16 STT plus structured extraction—consent, speakers, decisions, action items, PII, and HITL. Continue to Multi-Agent Travel Planner.