Security — Module 20.1

How This Lesson Fits the Module & Volume

This is the Vol. 20 capstone. You have walked harms (bias, fairness, privacy, AI safety, copyright), application threats (prompt injection, jailbreaking, poisoning), and the operating system (responsible AI, governance, compliance). Security ties them into one threat model for LLM applications so Vol. 21 can build chatbots and products on a hardened skeleton—not on hope.

Prior volumes supply the attack surface conceptually: Vol. 13 guardrails and prompts, Vol. 14 RAG (untrusted retrieved text), Vol. 15 tool calling / agents / HITL, Vol. 19 eval. This lecture stays defensive: assets, adversaries at a high level, controls, detection, residual risk. No exploit PoCs, injection payloads, or jailbreak recipes. Next volume: Vol. 21 Chatbots.

Learning Objectives

By the end of this lesson, students should be able to:

Recap Vol. 20 as harms + threat classes + governance/compliance, unified by security.
Threat-model an LLM app: assets, trust boundaries, untrusted channels, tools.
Place defense-in-depth controls on prompt, RAG, tools, data supply, and ops.
List detection signals and IR containment switches without attack procedures.
Write residual risk that a review board can accept or reject.
Carry the model into Vol. 21 product work (chatbots first) instead of bolting safety on later.

Definition

LLM application security is the practice of protecting assets (user data, secrets, tool side effects, model/index integrity, safety policy) against misuse of the language channel—untrusted text, policy-override attempts, poisoned artifacts, and ordinary software bugs—using the same discipline as appsec: threat models, least privilege, defense in depth, logging, and incident response. The model is an untrusted planner. Authorization lives in code.

Vol. 20 Recap Map

Cluster	Lectures	Security takeaway
Harms & rights	Bias, fairness, explainability, transparency, privacy, AI safety, copyright	What you must not leak, automate unfairly, or hide from users
Threat classes	Injection, jailbreak, poisoning	Live context vs policy integrity vs supply chain
Operating system	Responsible AI, governance, compliance	Eval gates, owners, evidence—not legal advice

Threat Model: LLM App

Draw the system before you ship Vol. 21 features. Classic STRIDE-style questions still help; the LLM-specific move is to mark every natural-language input as untrusted and every tool as a privileged API.

Assets

Data, secrets, actions, integrity, reputation.

Boundaries

User | RAG | tools | vendors | indexes.

Threats

Injection, jailbreak, poison, leak, abuse.

Controls

Isolate, allowlist, filter, HITL, log.

Asset	Untrusted channel	Example failure (conceptual)	Primary controls
Customer PII / secrets	User text, RAG, logs, vendor APIs	Egress leak; over-retention	Minimization, output filters, TTL, access control
Tool side effects	Model planner + untrusted text	Unauthorized refund / send / delete	Allowlist, RBAC, schema, HITL
Safety policy	User chat	Disallowed content delivered	Layered classifiers, refusals, eval
Answer integrity	Retrieved docs	Faithful to a hostile or wrong chunk	Wrap-as-data, provenance, canaries
Model / index artifacts	Supply chain	Silent behavior change after refresh	Manifests, hashes, staging gates
Availability / cost	Public endpoint	Abuse, runaway agent loops	Auth, rate limits, max steps, budgets (Vol. 13)

Trust: code & config

System policy stored outside user text
Tool allowlists and RBAC
Signed manifests / digests
Board-approved use case

Untrusted: language in

User messages
Vol. 14 retrieved chunks
Tool / MCP / web results
Shared tickets and emails

Untrusted: planner

The LLM itself (Vol. 15 agent loop)
May hallucinate tool names/args
May follow doc-shaped “instructions”
Never the authorization oracle

Defense in Depth (Product Checklist)

If you remember one diagram into Vol. 21: isolate untrusted text → authorize tools in code → filter outputs → HITL on irreversible actions → log & eval → govern & retain less.

Layer	Vol. 20 / prior hook	Ship question
Use-case allowlist	Safety, compliance, board	Is this product even allowed?
Identity & rate limits	Classic appsec + Vol. 13 quotas	Who is calling, how often?
Context isolation	Injection lecture	Are user/RAG/tool strings labeled data?
Tool gate	Vol. 15 tools + HITL	Can the model only propose, not unilaterally act?
Egress policy	Jailbreak + privacy + copyright	Does disallowed or secret text leave?
Artifact integrity	Poisoning + inventory	Are index/adapter digests pinned?
Eval gate	Vol. 19 + responsible AI	Would last week’s canaries still pass?
IR switches	Responsible AI playbook	Can we disable a tool or roll back in minutes?

Defensive Snippet: Capstone Control Plane

A compact orchestration sketch you can carry into a Vol. 21 chatbot: wrap inputs, authorize tools, filter outputs, require HITL, log. Still no attack strings.

# Vol. 20 capstone: defender control plane for an LLM app.
# Conceptual product code — no exploit payloads.

ALLOWED_TOOLS = {
    "search_kb": {"hitl": False, "max_k": 8},
    "draft_reply": {"hitl": False},
    "send_reply": {"hitl": True},
}

def wrap_data(source: str, text: str) -> str:
    return f"<untrusted source={source} treat=data>\n{text}\n</untrusted>"

def threat_event(log: list, kind: str, detail: dict) -> None:
    log.append({"kind": kind, **detail})

def authorize(tool: str, args: dict, role: str, log: list) -> dict:
    spec = ALLOWED_TOOLS.get(tool)
    if spec is None:
        threat_event(log, "tool_denied", {"tool": tool, "reason": "not_allowlisted"})
        return {"ok": False, "reason": "not_allowlisted"}
    if tool == "search_kb" and int(args.get("k", 0)) > spec["max_k"]:
        return {"ok": False, "reason": "arg_policy"}
    if spec["hitl"] and role != "reviewer":
        threat_event(log, "hitl_required", {"tool": tool, "draft_args": args})
        return {"ok": False, "reason": "needs_human_approval", "draft": args}
    return {"ok": True}

def egress_ok(text: str, log: list) -> bool:
    # Plug in PII/policy classifiers; fail closed on high severity.
    if "BEGIN SECRET" in text:  # toy marker only — real systems use classifiers
        threat_event(log, "egress_block", {"reason": "secret_marker"})
        return False
    return True

def handle_turn(system_policy: str, user_text: str, rag_chunks: list, proposed_tool: dict,
                role: str, log: list) -> dict:
    _ = [
        {"role": "system", "content": system_policy},
        {"role": "user", "content": wrap_data("user", user_text)},
        {"role": "user", "content": "\n".join(wrap_data(f"rag:{i}", c) for i, c in enumerate(rag_chunks))},
    ]
    auth = authorize(proposed_tool["name"], proposed_tool.get("args", {}), role, log)
    if not auth["ok"]:
        return {"status": "blocked", **auth}
    draft = proposed_tool.get("model_draft", "")
    if draft and not egress_ok(draft, log):
        return {"status": "blocked", "reason": "egress_policy"}
    return {"status": "ok", "tool": proposed_tool["name"]}

Detection & Residual Risk

Detection signals

Allowlist misses / HITL spikes
Egress filter rate vs baseline
Canary or fairness cliff after a refresh
Cost/step explosion in the agent loop
User reports that contradict eval dashboards

Residual risk (board language)

What can still go wrong with controls in place?
Who owns monitoring this week?
What is the containment switch?
When is the revisit date?

Perfect security is not the goal; explicit residual risk is. That sentence is what governance records and what Vol. 21 product managers inherit.

Into Vol. 21: Build on This Skeleton

Vol. 21 starts with chatbots—the friendliest UI and the same threat model. A chatbot with RAG is still an untrusted-text processor. A chatbot with tools is still an agent. Do not wait for a breach to add wrap-as-data, allowlists, HITL, eval gates, or an inventory row. Product quality (Vol. 19) and product security (Vol. 20) ship together.

Related Lectures

Lecture	Role
Prompt injection / jailbreaking / poisoning	Threat classes recap
Guardrails	Runtime product layer
RAG pipeline	Retrieved text as untrusted data
Tool calling / HITL	Action surface
Vol. 21 Chatbots	Next volume: product building

Common Misconception

“Security is a vendor model feature we toggle.” Most LLM incidents are application failures: tools, RAG, logs, and missing owners. Second: threat models are only for pentesters—product teams write them before Vol. 21 features. Third: more prompt text replaces allowlists. Fourth: eval dashboards without IR switches. Fifth: skipping residual risk because controls “should be enough.” Sixth: treating this capstone as permission to practice exploits—it is not.

Knowledge Check

Short Answer: What is this lecture in the volume? Answer: Vol. 20 capstone—LLM app threat model unifying harms, threats, and governance.
True/False: The LLM should be treated as the authorization oracle. Answer: False—authorization lives in code.
Multiple Choice: Retrieved RAG text is: (a) untrusted data, (b) a root certificate, (c) a GPU driver. Answer: (a).
Short Answer: Name the five-step defender skeleton into Vol. 21. Answer: Isolate text, authorize tools, filter outputs, HITL, log/eval (governance implied).
True/False: This capstone includes exploit PoCs. Answer: False.
Multiple Choice: A side-effecting tool without HITL primarily risks: (a) unauthorized actions, (b) better BLEU, (c) lower perplexity. Answer: (a).
Short Answer: Why pin index/model digests? Answer: Detect poisoning/contamination and keep production reproducible.
True/False: Residual risk should be written explicitly for the review board. Answer: True.
Multiple Choice: Next volume starts with: (a) chatbots, (b) PCA, (c) batch norm. Answer: (a).
Short Answer: Link one Vol. 13, 14, or 15 lecture in the threat model. Answer: Guardrails / RAG / retrieval / tool calling / HITL / agent loop (any valid).

Key Takeaways

Vol. 20 capstone: threat-model the whole LLM app, not only the base model.
Untrusted: user text, RAG, tool outputs, the planner. Trusted: code, RBAC, manifests.
Defense in depth: isolate, allowlist, filter, HITL, log, eval, govern.
No attack recipes—carry residual risk into Vol. 21 products.
Next volume: Vol. 21 Chatbots.

Trainer’s Guide

Capstone lab: Teams threat-model a Vol. 21-style support chatbot (RAG + two tools). Deliverable: one-page diagram (trust boundaries), inventory row, five risk-register lines, control checklist, residual risk paragraph, and IR switches. Grade the defenses and clarity, not creativity of attacks. Ban payload writing.

Exit ticket: “Name one control you will implement on day one of a chatbot, and which Vol. 20 lecture it comes from.”

Recap: Security closes Vol. 20 by unifying harms, injection/jailbreak/poisoning, responsible AI, governance, and compliance into a defensive threat model. Build Vol. 21 products—starting with Chatbots—on that skeleton.