Model Poisoning — Module 20.1

How This Lesson Fits the Module & Volume

Prompt injection and jailbreaking abuse a live context window. Model poisoning (and its close cousins: corpus poisoning and supply-chain tampering) abuses the data that builds or grounds the system—pretraining mixtures, fine-tunes, RLHF labels, RAG indexes (Vol. 14), or third-party adapters. The model then misbehaves even for honest users and clean prompts.

This is a defender lecture: provenance, signing, staging evals, canary tasks, and incident rollback. It does not describe how to craft poisoned samples or backdoors. After integrity of the model and corpus, Vol. 20 turns to organizational practice: responsible AI, governance, and compliance.

Learning Objectives

By the end of this lesson, students should be able to:

Define poisoning as tampering with training, fine-tune, or grounding data/supply chain.
Map threat surfaces: pretrain, SFT/RLHF, RAG index, third-party adapters.
Specify provenance, hashing, access control, and staging gates.
Use canary evals and safety regression to detect integrity failures.
Plan rollback / isolation when a data refresh looks hostile or accidental.
Relate poisoning to Vol. 14 knowledge bases and Vol. 20 bias/safety outcomes.

Definition

Model poisoning is the threat class in which an adversary (or a sloppy pipeline) alters the data or artifacts that define model behavior—training sets, preference labels, fine-tune checkpoints, LoRA adapters, or the documents in a retrieval index—so that downstream predictions, refusals, or RAG answers systematically deviate from the owner’s intent. Accidental contamination (wrong labels, scraped malware pages, unreviewed vendor dumps) is treated with the same integrity controls even when there is no attacker. This curriculum covers detection and hardening, not attack construction.

Threat Surfaces (High Level)

Surface	What can go wrong	Product symptom
Pretrain / continued pretrain	Unvetted web or partner dumps	Broad capability or safety drift
SFT / preference data	Bad labels, insider tampering	Refusal collapse; biased answers (bias)
Third-party weights / adapters	Unsigned downloads, typo-squatted repos	Unexpected tool use or policy holes
RAG / KB (Vol. 14)	Poisoned or vandalized documents	Faithful-but-wrong answers; indirect injection fuel
Eval sets themselves	Canaries deleted or “fixed” to pass	False confidence at ship time

RAG corpus poisoning is especially practical: you may never retrain the LLM, yet Vol. 14 indexing silently teaches the product new “facts.” That is why retrieval must stay data (injection lecture) and why the index needs provenance like a model artifact.

Defender Architecture: Integrity Pipeline

Source

Who/what produced this data?

Seal

Hash, sign, access control.

Stage

Eval + canaries offline.

Ship

Promote or rollback.

Provenance

Dataset manifest: owner, license, date
Least-privilege write to the index / train bucket
Human review for high-impact corpora

Integrity

Content hashes; signed checkpoints
Pin adapter / model digests in config
Alert on unexpected file mutations

Detection

Canary prompts that must stay stable
Safety + fairness regression (Vol. 19 / 20)
Retrieval spot-checks after index updates

Control	Question it answers
Model / dataset inventory	What is in production, and from where? (see governance)
Staging vs production indexes	Did canaries fail before users saw the refresh?
Separate eval custody	Can someone who edits train data also silently edit the test?
Rollback runbook	Can we restore the last signed artifact in minutes?

Defensive Snippet: Manifest, Hash, Eval Gate

The following is a policy-as-code sketch: refuse to promote a dataset or index snapshot unless the manifest matches on-disk hashes and a small canary eval still passes. Swap the toy canary_eval for your real Vol. 19 safety / groundedness suite.

# Defender-only: dataset / index integrity gate. No poisoning recipes.

import hashlib
import json
from pathlib import Path

def sha256_file(path: Path) -> str:
    h = hashlib.sha256()
    with path.open("rb") as f:
        for chunk in iter(lambda: f.read(1 << 20), b""):
            h.update(chunk)
    return h.hexdigest()

def load_manifest(path: Path) -> dict:
    return json.loads(path.read_text(encoding="utf-8"))

def verify_snapshot(root: Path, manifest: dict) -> list[str]:
    errors = []
    for rel, expected in manifest["files"].items():
        p = root / rel
        if not p.is_file():
            errors.append(f"missing:{rel}")
            continue
        digest = sha256_file(p)
        if digest != expected:
            errors.append(f"hash_mismatch:{rel}")
    return errors

def canary_eval(run_fn) -> dict:
    """Stable tasks the product must not regress after a data refresh."""
    cases = manifest_canaries()  # owned by safety; not editable by data producers
    fail = 0
    for item in cases:
        pred = run_fn(item["input"])
        if pred != item["expected"]:
            fail += 1
    return {"n": len(cases), "fail": fail, "ok": fail == 0}

def promote_if_clean(root: Path, manifest_path: Path, run_fn) -> dict:
    man = load_manifest(manifest_path)
    errors = verify_snapshot(root, man)
    if errors:
        return {"promoted": False, "reason": "integrity", "errors": errors}
    ev = canary_eval(run_fn)
    if not ev["ok"]:
        return {"promoted": False, "reason": "canary_regression", "eval": ev}
    return {"promoted": True, "manifest_id": man["id"], "eval": ev}

def manifest_canaries():
    return [{"input": "health_canary_1", "expected": "stable_refusal_or_answer"}]

Detection Signals (No Attack Steps)

Watch for

Hash / signature mismatch vs last approved manifest
Canary or safety eval cliff after a data or adapter bump
Sudden bias metric shift (fairness)
Index growth from an unexpected publisher
Unsigned model download in CI logs

Response (process)

Freeze promotions; rollback to last signed artifact
Quarantine the suspect snapshot; do not “fix forward” blindly
Incident ticket under responsible AI
Review who had write access (governance)

Related Lectures

Lecture	Role
Knowledge base / indexing	Corpus integrity = RAG integrity
Bias / AI safety	Downstream harms of bad data
Hallucination tests	Catch unfaithful or hostile grounding
Prompt injection	Poisoned docs also feed indirect injection
Responsible AI	Next: principles to practices

Common Misconception

“We use a famous base model, so poisoning does not apply.” Your fine-tune, adapter, and RAG index are still in the blast radius. Second: poisoning only means a cinematic training-set attack—accidental contamination and unsigned downloads are the common cases. Third: “Eval passed last month” without pinning the eval set and re-running after every data refresh. Fourth: treating the wiki as trusted because employees wrote it—wikis get vandalized; use review + hashes. Fifth: conflating poisoning with jailbreaking; one tampers with artifacts, the other tampers with a single session.

Knowledge Check

Short Answer: What is model poisoning as a threat class? Answer: Tampering with training, fine-tune, or grounding data/artifacts so behavior systematically changes.
True/False: A RAG index can be a poisoning surface without retraining the LLM. Answer: True.
Multiple Choice: A content hash on a snapshot primarily detects: (a) integrity changes, (b) BLEU, (c) GPU thermal throttle. Answer: (a).
Short Answer: Why separate custody of eval/canary sets from data producers? Answer: So the test cannot be silently edited to hide poisoning or contamination.
True/False: This lecture teaches how to build poisoned backdoor samples. Answer: False.
Multiple Choice: After a canary cliff you should first: (a) rollback / freeze promote, (b) increase temperature, (c) drop the system prompt. Answer: (a).
Short Answer: Name one Vol. 14 lecture tied to corpus integrity. Answer: Knowledge base / indexing / RAG / retrieval (any valid).
True/False: Unsigned third-party adapters are an integrity risk. Answer: True.
Multiple Choice: Next lecture: (a) responsible AI, (b) k-means, (c) RoPE. Answer: (a).
Short Answer: How does poisoning differ from prompt injection? Answer: Poisoning tampers with stored artifacts/data supply; injection hijacks a live context window.

Key Takeaways

Poisoning / contamination = integrity failure of data, indexes, or weights.
Manifests, hashes, access control, staging canaries, and rollback are the defense.
RAG corpora need the same discipline as model checkpoints.
No attack recipes—detect, isolate, roll back, review access.
Next: Responsible AI.

Trainer’s Guide

Lab (defensive only): Students write a manifest + hash verifier and a three-item canary gate. Simulate an accidental file mutation (edit a dummy corpus file) and show promote-blocked. Optional: role-play an incident standup (who freezes the index, who owns rollback) without constructing hostile samples.

Whiteboard: Supply chain: vendor dump → review → signed snapshot → staging eval → production index. Mark where Vol. 14 retrieval and Vol. 20 bias evals attach.

Recap: Model and corpus poisoning are supply-chain integrity problems. Seal artifacts, gate promotions with canaries, and roll back fast. Organizational practice continues in Responsible AI.