AI Safety — Module 20.1

How This Lesson Fits the Module & Volume

Bias and fairness ask for whom. Privacy asks what must not leak. AI safety asks whether the system can cause unacceptable harm even when metrics look fine: misuse, overreliance, and a gap between capability and control. This lecture stays high-level. Later siblings—prompt injection, jailbreaking, model poisoning, security—go deeper without turning this page into an attack manual.

Vol. 19 benchmarks almost never score these risks. Governance and responsible AI are how orgs assign owners after you name the hazards.

Learning Objectives

By the end of this lesson, students should be able to:

Define AI safety (this module) as harm prevention under intended and unintended use—high level.
Distinguish misuse, overreliance/automation bias, and accidents from Vol. 19 quality failures.
Explain capability vs control: what the model can do vs what your product allows and monitors.
List layered controls: spec, eval, product UX, human oversight, monitoring—without exploit steps.
Write a small allow/deny policy gate sketch for a tool-using assistant.
Know when to escalate to security, privacy, copyright, and governance lectures.

Definition

AI safety (here) is the practice of reducing unacceptable harm from AI systems across their lifecycle: specifying what must not happen, evaluating for those failures, constraining deployment, and monitoring live use. It includes misuse (a person tries to cause harm with the system), overreliance (people trust fluent output more than they should), and control failures (capability exists without reliable shutoff, scoping, or oversight). It is broader than model “alignment” slogans and narrower than all of ethics. This lecture does not teach how to jailbreak, weaponize, or evade safeguards.

Hazard Families (High Level)

Family	What it looks like in a product	Not the same as
Misuse	Requests for fraud, malware, harassment, or other clearly harmful assistance	A curious student asking how a lock works in a locksmith class
Overreliance	Users accept hallucinated doses, legal cites, or financial advice because the tone is confident	Vol. 19 accuracy alone
Accident / spec miss	Tool calls the wrong API, deletes data, emails the wrong tenant	A low BLEU score
Capability–control gap	The model can draft a risky plan; the product still exposes unconstrained tools	“SOTA on MMLU”
Sociotechnical harm	Bias, privacy leaks, copyright issues compounding into real-world damage	A single metric dashboard

Capability growth without matching evals, access control, and human processes is the core engineering story. Your job is rarely “make the model weaker”; it is “scope what the product can do, for whom, with what evidence.”

Capability vs Control

Capability

What the model + tools could do (draft, retrieve, call APIs).
Grows with better models, longer context, more plugins.
Vol. 19 mostly measures capability slices.

Control

Auth, allowlists, rate limits, human confirm, logging.
Refusal/redirect policy for out-of-scope harm.
Kill switches, staged rollout, incident response.

Overreliance controls

UX: uncertainty, citations, “not advice”.
Force confirm on irreversible actions.
Train users; measure override rates.

Layered safety buys

No single filter has to be perfect
Product constraints even if the model is general
A paper trail for governance

A single chat filter does not buy

Tool-use safety (the API still fires)
Protection against overreliance on fluent wrong answers
An excuse to skip privacy, fairness, and copyright

Policy Gate Sketch (Defensive, High Level)

The snippet is a product control pattern: classify intent at a coarse level and require human confirmation before irreversible tools. It is not a recipe to probe or bypass anyone else’s safeguards.

# Defensive product policy — high level, not an attack or jailbreak guide.
ALLOWED_TOOLS = {"search_kb", "create_draft_reply"}
IRREVERSIBLE_TOOLS = {"send_email", "delete_record", "refund"}
DISALLOWED_INTENTS = {"malware", "fraud_assistance", "violent_wrongdoing"}  # coarse labels

def safety_gate(user_intent: str, tool: str, human_confirmed: bool) -> str:
    if user_intent in DISALLOWED_INTENTS:
        return "refuse"  # explain policy; do not provide harmful how-to
    if tool not in ALLOWED_TOOLS and tool not in IRREVERSIBLE_TOOLS:
        return "refuse_unknown_tool"
    if tool in IRREVERSIBLE_TOOLS and not human_confirmed:
        return "require_human"
    return "allow"

# Overreliance UX flags (pair with Vol. 19 hallucination tests):
def user_facing_banner(groundedness_ok: bool, domain: str) -> str:
    if domain in {"medical", "legal", "financial"}:
        return "Not professional advice. Confirm with a qualified human."
    if not groundedness_ok:
        return "Low evidence. Treat this answer as unverified."
    return ""

# Capability vs control checklist (ops):
# [ ] Intended use written (transparency card)
# [ ] Tools scoped + authenticated
# [ ] Eval for refusals + overreliance scenarios (not only MMLU-style)
# [ ] Monitoring + incident owner (governance)
# [ ] Privacy + copyright constraints still apply
print(safety_gate("refund_question", "refund", human_confirmed=False))
print(user_facing_banner(False, "legal"))

Related Lectures

Lecture	Role
Prompt injection / Jailbreaking	Adversarial control failures (later; still not how-to attacks here)
Model poisoning	Training-time integrity
Security	Auth, isolation, secrets
Privacy	Leakage as a safety-relevant harm
Responsible AI / Governance	Owners, gates, incidents
Hallucination tests (Vol. 19)	Overreliance on ungrounded answers

Common Misconception

“Safety = a polite refusal string.” Tools, logs, and humans decide real harm. Second: high benchmark scores imply safety. Third: overreliance is the user’s fault only—UX and confirmations are engineering. Fourth: this lecture should include jailbreak recipes; it must not. Fifth: capability reductions are the only control (scoping the product is usually better). Sixth: safety work replaces fairness, privacy, or copyright review.

Knowledge Check

Short Answer: Name three high-level AI safety concerns in this lecture. Answer: Misuse, overreliance, and capability vs control (also accidents/spec misses).
True/False: Vol. 19 public benchmarks usually measure misuse and overreliance well. Answer: False.
Multiple Choice: Overreliance is: (a) trusting fluent output more than evidence warrants, (b) using FP16, (c) a BLEU variant. Answer: (a).
Short Answer: What is the capability–control gap? Answer: The model/tools can do more than the product reliably constrains, monitors, or oversees.
True/False: This lecture teaches how to jailbreak production models. Answer: False—high-level only; no attack procedures.
Multiple Choice: Irreversible tool calls should typically: (a) require human confirmation, (b) auto-fire always, (c) skip logs. Answer: (a).
Short Answer: Give one overreliance UX control. Answer: Any of: uncertainty banners, citations, “not advice,” forced confirm, human handoff.
True/False: A single chat filter is sufficient control for tool-using agents. Answer: False.
Multiple Choice: Misuse (high level) means: (a) someone tries to cause harm with the system, (b) dropout is too high, (c) the learning rate is small. Answer: (a).
Short Answer: Which later sibling lectures cover adversarial prompt issues? Answer: Prompt injection and jailbreaking (and related security).

Key Takeaways

AI safety here is harm reduction: misuse, overreliance, accidents, and control of capability.
Vol. 19 quality ≠ safety; scope the product, not only the model card slogan.
Layer specs, evals, UX, humans, and monitoring; keep this page non-operational for attacks.
Irreversible tools need confirmation; fluent answers still need evidence.
Next: Copyright — training data, outputs, and licensing trade-offs.

Trainer’s Guide

Lab: Students write a one-page intended-use + disallowed-use list for a tool-using support bot, then implement the gate sketch. Role-play overreliance: a confident wrong refund policy. No live probing of external model defenses.

Whiteboard: Two bars—Capability vs Control—growing at different speeds. Arrows to Privacy, Fairness, and later Prompt injection as specific control failures.

Recap: AI safety is high-level harm control—misuse, overreliance, and keeping capability inside product constraints. Do not confuse it with leaderboards. Continue to Copyright.