Healthcare AI — Module 21.1

How This Lesson Fits the Module & Volume

Research assistants made citation a product feature. Healthcare AI is the first Vol. 21 regulated vertical: the same RAG + LLM stack, but health data, clinical workflow, and safety-critical claims change the architecture. Vol. 20 privacy, compliance, and responsible AI are no longer optional chapters—they are the product constraints. Vol. 15 HITL is the default, not a maturity luxury.

This lecture is educational engineering, not medical advice, not a device claim, and not a license to practice medicine. Students learn product patterns (task allowlists, clinician sign-off, audit logs, minimization) so they do not ship a chatbot that pretends to diagnose. Finance and legal lectures repeat the same regulated template next.

Learning Objectives

By the end of this lesson, students should be able to:

State explicitly that healthcare AI products here are engineering patterns, not medical advice or clinical devices.
Choose in-scope tasks (drafting, extraction, navigation) vs out-of-scope (diagnose, prescribe, treat as the product claim).
Place clinician HITL and audit logs on every high-impact artifact.
Apply Vol. 20 privacy/compliance themes: minimization, purpose, retention, vendor no-train.
Reuse Vol. 19 hallucination tests on clinical text without treating faithfulness as safety.
Know when to stop and escalate to qualified clinical, privacy, and regulatory specialists.

Definition

Healthcare AI (in this curriculum) means AI features used in care delivery, administration, or patient communication that process health-related information. A regulated product pattern here is: narrow task allowlist + human clinician (or qualified staff) in the loop + immutable audit + privacy/compliance controls + no claim that the model itself diagnoses, prescribes, or replaces a licensed professional. PHI / health data is personal data with extra sensitivity—prompts, RAG chunks, and logs are still processing (Vol. 20). Engineers implement capabilities; clinicians and counsel decide what is lawful and clinically acceptable.

Not Medical Advice

Nothing on this page is medical, diagnostic, or treatment advice. Law names and “HIPAA-style” labels are teaching themes, incomplete, and not a certification. Real products need qualified clinical safety review, privacy/legal counsel, and often formal regulatory pathways. Do not ship “our app diagnoses you” from a lecture.

In-Scope vs Out-of-Scope Product Claims

Usually in-scope (with HITL)	Usually out-of-scope as the product claim
Draft visit summaries / after-visit instructions for clinician edit	“You have X; take Y” consumer diagnosis/prescription
Extract structured fields from notes the org already stores	Autonomous triage that discharges or refuses care
Navigate a clinician-approved knowledge base with citations	Uncited treatment recommendations to patients
Admin: scheduling language, benefits explainer labeled non-binding	Eligibility or coverage determination with no human
Translation / plain-language rewrite of clinician-approved text	Silent model-to-patient advice without disclosure

The line is the claim and the side effect, not the model brand. A research-assistant stack (previous lecture) behind a “symptom checker that tells you what you have” is still out of policy. A documentation copilot that cannot sign itself is in the regulated pattern.

Regulated Pattern: HITL + Audit + Privacy

HITL (Vol. 15)

Clinician edit + sign before release
Escalate ambiguity / red-flag language
Sampled review even after maturity
Patient-facing copy: “this is AI-drafted” (Vol. 20 transparency)

Audit logs

Who viewed / drafted / edited / signed
Model version + prompt template ID
Purpose tag + retention TTL
Not a second unprotected PHI copy

Privacy / compliance (Vol. 20)

Minimization: do not paste extra charts
Vendor BAA / no-train flags
RAG ACLs: break-glass, not “all clinicians see all”
Delete/export considers embeddings

Narrow allowlist buys

Clear eval (Vol. 19) on draft quality, not “accuracy of disease”
Smaller regulatory and safety blast radius
Honest marketing: assist, do not replace

Oracle chatbot costs

Hallucinated citations about drugs (Vol. 19)
Bias/fairness harms in triage (Vol. 20 fairness)
PHI in vendor logs you cannot delete

Documentation Copilot Sketch (Not a Diagnostic Device)

Illustrative flags for a system you operate. Not a medical device software spec. Not advice to patients or clinicians about care.

# HITL + audit for a CLINICAL DOCUMENTATION assistant — not a diagnostic product.
# Educational engineering. Not medical advice. Not a device claim.

from datetime import datetime, timezone
import hashlib

ALLOWED_TASKS = {
    "draft_visit_summary",
    "extract_codes_draft",
    "plain_language_rewrite_approved_text",
}
ADVICE_MARKERS = ("you have", "you should take", "i diagnose", "stop your medication")

def classify_task(intent: str) -> str:
    if intent not in ALLOWED_TASKS:
        return "refuse_out_of_scope"
    return intent

def clinician_gate(task: str, draft: str) -> dict:
    low = draft.lower()
    if any(m in low for m in ADVICE_MARKERS):
        return {
            "ok": False,
            "reason": "sounds_like_advice_rewrite_as_questions_for_clinician",
        }
    return {
        "ok": True,
        "status": "pending_clinician_review",
        "task": task,
        "disclaimer": (
            "Draft only. Not a diagnosis, prescription, or care plan. "
            "Licensed clinician must edit and sign."
        ),
    }

def audit_event(actor_id: str, action: str, record_id: str, model_id: str, extra=None):
    return {
        "ts": datetime.now(timezone.utc).isoformat(),
        "actor_id": actor_id,
        "action": action,  # view | draft | edit | sign | export
        "record_hash": hashlib.sha256(record_id.encode()).hexdigest()[:16],
        "model_id": model_id,
        "purpose": "care_documentation",
        "ttl_days": 365,  # stub — counsel/clinical policy sets real retention
        "extra": extra or {},
    }

# Eval (Vol. 19): groundedness vs clinician-approved KB; hallucination tests;
# never claim "diagnostic accuracy" for an out-of-scope symptom-oracle task.

Related Lectures

Lecture	Role
Research assistants	Citation + RAG faithfulness reused here
Document AI	Extraction sibling; still needs PHI controls
HITL	Clinician sign-off primitive
Privacy / Compliance	Minimization, rights, risk tiers
Fairness / Bias	Unequal error across groups
Hallucination tests	Fake citations are safety incidents
Finance AI	Next: same regulated template, money instead of PHI

Common Misconception

“A disclaimer in the footer makes autonomous diagnosis OK.” Product claim + side effect matter more than a footer. Second: HIPAA-style themes are satisfied by regex redaction alone (Vol. 20: redaction \(\neq\) anonymization). Third: RAG over PubMed means the model is a doctor. Fourth: faithfulness to a wrong guideline is safety. Fifth: audit logs can live in the same chat table as PHI with open access. Sixth: this lecture authorizes you to give medical advice or skip counsel.

Knowledge Check

Short Answer: Is this lecture medical advice? Answer: No—it is educational engineering only.
True/False: A healthcare chatbot should claim to diagnose and prescribe as its product. Answer: False—that claim is out of scope here.
Multiple Choice: Default control for high-impact clinical artifacts: (a) clinician HITL + sign, (b) higher temperature, (c) skip logs. Answer: (a).
Short Answer: Name two Vol. 20 lectures that constrain health data in prompts/RAG/logs. Answer: Privacy and compliance (also responsible AI / security acceptable).
True/False: Prompt logs cannot contain PHI. Answer: False—they often do.
Multiple Choice: Invented drug citations in a care draft are: (a) a hallucination/safety incident, (b) a good BLEU win, (c) mixed precision. Answer: (a).
Short Answer: What should an audit event typically record besides the text itself? Answer: Actor, action, time, record id/hash, model version, purpose (any reasonable subset).
True/False: A footer disclaimer alone makes autonomous patient diagnosis acceptable. Answer: False.
Multiple Choice: Vendor training on clinical chats without agreement is primarily a: (a) privacy/compliance failure, (b) Cosine similarity bug, (c) BLEU feature. Answer: (a).
Short Answer: When must engineers escalate instead of “just shipping”? Answer: Clinical safety, device/regulatory pathway, lawful basis, or any “we diagnose” marketing claim—call specialists.

Key Takeaways

Healthcare AI in this volume = assistive, HITL, audited, privacy-constrained—not a doctor in a box.
Product claim matters: drafting/extraction \(\neq\) diagnose/prescribe/treat.
Vol. 20 privacy/compliance and Vol. 19 hallucination tests are part of the architecture.
This page is not medical advice; escalate real regulatory questions.
Next: Finance AI — same pattern for money and credit.

Trainer’s Guide

Lab: Students map a toy “after-visit summary drafter”: data flow, ACL, TTL, clinician sign state machine, and three refused user stories (“tell me what disease I have”). No real PHI; synthetic notes only. No clinical recommendations to classmates.

Whiteboard: Draw allowlist vs oracle chatbot. Arrow fake citation \(\to\) incident, not “retry the prompt.” Repeat: not medical advice.

Recap: Healthcare AI products here are narrow, clinician-HITL, audited, and privacy-aware—never a substitute for licensed care. Continue to Finance AI.