Explainability — Module 20.1

How This Lesson Fits the Module & Volume

After bias and fairness, reviewers ask why this score or this sentence. Explainability is how we produce reasons that humans can check—feature attributions for tabular models, citations and grounding for LLMs. It is not a substitute for fairness metrics, and a fluent chain-of-thought is not automatically a true causal story.

Next lecture, Transparency, is what you disclose (model cards, data cards). Explainability is what you can show for a decision or answer. Grounding links back to Vol. 19 hallucination tests and Vol. 14 RAG. Later AI safety still applies when explanations look confident and the action is unsafe.

Learning Objectives

By the end of this lesson, students should be able to:

Define explainability vs interpretability vs transparency in this module’s vocabulary.
Contrast global vs local explanations and intrinsic vs post-hoc methods.
Describe feature importance and SHAP-style additive attributions conceptually (not as proof of fairness).
Specify LLM citation/grounding as an explanation pattern and know when it fails.
Implement a small permutation-importance or citation-check sketch in Python.
List failure modes: unstable attributions, proxy features, unfaithful rationales.

Definition

Explainability is the ability to produce human-usable reasons for a model output: which inputs moved a score, which retrieved passages support a claim, or which rule fired. Interpretability usually means the model is simple enough to inspect directly (linear weights, short trees). Transparency (next lecture) is organizational disclosure—cards, docs, user notices—not a SHAP plot. SHAP and permutation importance estimate feature influence under stated assumptions; they do not prove causality, legality, or fairness. For LLMs, a citation is an explanation only if the cited span actually supports the claim (Vol. 19 groundedness).

A Map of Explanation Styles

Style	Question	Typical tool
Intrinsic	Can a person read the model itself?	Linear/logistic weights, shallow trees, rule lists
Global post-hoc	What matters on average?	Permutation importance, global SHAP summaries
Local post-hoc	Why this row / this token?	Local SHAP/LIME-style attributions, attention (weak)
Counterfactual	What smallest change flips the decision?	Actionable recourses (use with legal/product care)
LLM grounding	Which sources back this sentence?	Citations to retrieved chunks; claim–span checks
Process explanation	Which tool/policy ran?	Trace logs, guardrail IDs, human review stamps

Attention weights and chain-of-thought traces are sometimes useful and often unfaithful: the model can narrate a path it did not use. Treat them as hypotheses to verify, not as audit evidence.

Feature Importance and SHAP (Conceptual)

Permutation importance

Shuffle one feature; watch metric drop.
Model-agnostic; needs a labeled holdout.
Correlated features share credit poorly.

SHAP-style attributions

Idea: each feature gets a contribution that adds up to (prediction − baseline).
Inspired by Shapley values from cooperative game theory.
Local plots explain one row; beeswarm/bar plots summarize many.
Computationally heavy; still not causal; still not a fairness certificate.

LLM citations / grounding

Require chunk IDs or URLs next to claims.
Verify span entailment (Vol. 19 hallucination tests).
“According to the docs” without a checkable quote is theater.

Explanations help when

Debugging proxies and leakage-like shortcuts
Supporting a human reviewer (not replacing them)
Showing users a source for RAG answers

They fail when

Stakeholders treat SHAP as legal or causal proof
Rationales are fluent but unfaithful
The real driver is a feature you never logged

Practical Sketches: Tabular Importance + Citation Check

# 1) Model-agnostic permutation importance (sklearn). Conceptual cousin of SHAP.
#    SHAP libraries estimate additive local contributions; permutation estimates
#    global metric sensitivity. Neither proves fairness (see bias.html / fairness.html).
from sklearn.inspection import permutation_importance

# r = permutation_importance(model, X_val, y_val, n_repeats=10, random_state=0)
# for name, mean, std in zip(feature_names, r.importances_mean, r.importances_std):
#     print(f"{name:20s}  delta={mean:.4f} ± {std:.4f}")

# Toy local "additive" explanation sketch (NOT a full SHAP solver):
baseline = 0.40
contribs = {"tenure": 0.12, "balance": -0.05, "device": 0.03}  # invented demo numbers
local_score = baseline + sum(contribs.values())
assert abs(local_score - (baseline + 0.10)) < 1e-9
print("prediction ≈ baseline + sum(feature contribs) =", local_score)

# 2) LLM grounding: a citation is valid only if the claim is supported by the span.
def citation_ok(claim: str, span: str, entailment_fn) -> bool:
    """entailment_fn(claim, span) -> True if span supports claim (NLI / human)."""
    if not span.strip():
        return False
    return bool(entailment_fn(claim, span))

# Policy: refuse to show a footnote if citation_ok is False.
# Do not treat chain-of-thought text as evidence unless independently checked.

Product policy stub: high-stakes tabular decisions get a local attribution plus a human review path; RAG answers get claim-level citations; marketing copy may not quote SHAP values as “the reason we denied you” without legal review. See transparency for what goes on the card.

Related Lectures

Lecture	Role
Bias / Fairness	Explanations can surface proxies; they do not define fairness
Transparency	Cards and user disclosure vs per-decision reasons
Privacy	Explanations can leak features or retrieved PII
AI Safety	Confident rationale ≠ safe action
Hallucination tests (Vol. 19)	Groundedness / attribution checks
RAG (Vol. 14)	Where citations should point

Common Misconception

“SHAP proves the model is fair.” Attributions describe influence under a baseline, not group justice. Second: chain-of-thought is a faithful explanation of an LLM. Third: attention = explanation. Fourth: a citation that does not contain the claim. Fifth: showing users raw feature values that are PII. Sixth: skipping Vol. 19 hallucination tests because the UI has a nice footnote component.

Knowledge Check

Short Answer: How does explainability differ from transparency in this module? Answer: Explainability = reasons for an output; transparency = disclosure artifacts (cards, notices).
True/False: SHAP-style values prove a decision is legally fair. Answer: False.
Multiple Choice: Permutation importance primarily estimates: (a) global metric sensitivity to shuffling a feature, (b) GPU temperature, (c) BLEU. Answer: (a).
Short Answer: What do SHAP-style attributions try to make add up? Answer: Feature contributions summing to prediction minus a baseline.
True/False: Chain-of-thought is always a faithful account of model computation. Answer: False—it can be unfaithful.
Multiple Choice: An LLM citation is a real explanation only if: (a) the span supports the claim, (b) the font is blue, (c) the model says “trust me.” Answer: (a).
Short Answer: Name one reason correlated features break naive importance. Answer: Credit is split or stolen; shuffling one feature leaves the signal in another.
True/False: Interpretability usually means the model itself is inspectable (e.g. linear weights). Answer: True.
Multiple Choice: Grounding checks belong with: (a) Vol. 19 hallucination/attribution tests, (b) only Vol. 02 tensors, (c) CSS animations. Answer: (a).
Short Answer: Why can explanations create a privacy issue? Answer: They may reveal sensitive features or retrieved PII to users or logs.

Key Takeaways

Explainability = per-output reasons; transparency = what you publish about the system.
Permutation importance and SHAP-style additive attributions are influence estimates, not fairness proofs.
LLM explanations that matter are checkable citations/grounding, not unfaithful rationales.
Use explanations to debug and to support humans; verify them like any other eval.
Next: Transparency — model cards, data cards, disclosure.

Trainer’s Guide

Lab: Fit a tiny sklearn model; compute permutation importance. Optionally demo a SHAP summary if the environment has it—stress assumptions. Then take three RAG answers: one well-cited, one cited-but-unsupported, one fluent CoT with no sources. Score with a claim–span checklist.

Whiteboard: Split “why this score” vs “what we disclose on the card” vs “is this fair.” Draw a false-citation trap.

Recap: Explainability produces checkable reasons—feature attributions conceptually, citations in practice for LLMs. It does not replace fairness or safety. Continue to Transparency.