Prompt Evaluation — Module 13.1

How This Lesson Fits the Module & Volume

Module 13.1 taught techniques from basic prompting through chaining and guardrails. Prompt evaluation is how you know a change helped: fixed test sets, automatic checkers, human rubrics, and regression gates before shipping.

This closes Module 13.1 and hands off to Module 13.2—Prompt Structure—where writing craft deepens atop measurable quality loops.

Learning Objectives

By the end of this lesson, students should be able to:

Define prompt evaluation and why vibe-checks fail in production.
Build a golden set of inputs with expected properties or labels.
Combine automatic metrics (format, exact match, embedding similarity) with human rubrics.
A/B compare prompt versions under the same model and decoding settings.
Track safety and refusal quality alongside helpfulness.
Wire evals into CI or release checklists.

Definition

Prompt evaluation is the systematic measurement of prompt (and pipeline) quality on a representative set of cases using automated checks and/or human judgments, so teams can compare versions, catch regressions, and ship with evidence instead of anecdotes.

Evaluation Loop

Cases

Golden + edge set.

Run

Fixed model/params.

Score

Auto + human.

Decide

Ship, revise, or roll back.

Metric type	Examples	Best for
Format / schema	Valid JSON, required keys	JSON / tools
Exact / set match	Labels, IDs, enums	Classification
Overlap / similarity	BLEU-ish, embeddings	Paraphrase-ish tasks
LLM-as-judge	Rubric scores	Open-ended quality
Safety	Attack success rate, PII leaks	Guardrails

Must-have cases

Happy paths
Ambiguous inputs
Adversarial / injection

Control variables

Model ID + version
Temperature / top-p
Prompt version hash

Release bar

No format regression
Safety rate within SLO
Helpfulness ≥ baseline

Practical Mini Eval Harness

import json
from pathlib import Path

cases = json.loads(Path("evals/tickets_v1.json").read_text())
# each case: {"id", "input", "expect": {"priority": "high"}, "must_refuse": false}

def score_case(case, predict_fn):
    out = predict_fn(case["input"])  # returns dict or text
    results = {"id": case["id"]}
    try:
        data = out if isinstance(out, dict) else json.loads(out)
        results["valid_json"] = True
        results["priority_ok"] = data.get("priority") == case["expect"]["priority"]
    except Exception:
        results["valid_json"] = False
        results["priority_ok"] = False
    if case.get("must_refuse"):
        results["refused"] = is_refusal(out)
    return results

rows = [score_case(c, run_prompt_v3) for c in cases]
print("json_rate", sum(r["valid_json"] for r in rows) / len(rows))
print("priority_acc", sum(r.get("priority_ok") for r in rows) / len(rows))
# Gate: assert json_rate >= 0.95 and priority_acc >= baseline - 0.02

Strengths

Turns prompting into engineering
Catches silent regressions
Aligns product + safety goals

Tradeoffs

Eval sets need curation
Judges can be biased/noisy
Overfitting to the golden set

Common Misconception

“If it looked good in the playground, we’re done.” Playground vibes ignore distribution shift, format failures at scale, and safety edge cases. Treat prompts like code: version them, evaluate them, and only then promote—especially before Module 13.2 craft changes that feel better but measure worse.

Knowledge Check

Short Answer: What is prompt evaluation? Answer: Systematic measurement of prompt/pipeline quality on representative cases.
True/False: Changing temperature between A/B prompt tests is fine without noting it. Answer: False—control decoding settings.
Multiple Choice: Schema validity rates best evaluate: (a) structured JSON outputs, (b) GPU clock speed, (c) font kerning. Answer: (a).
Short Answer: Name three case types for a golden set. Answer: Happy path, ambiguous, and adversarial/injection (or equivalent).
True/False: Safety metrics belong in the same release bar as helpfulness. Answer: True.
Multiple Choice: LLM-as-judge is useful for: (a) open-ended quality rubrics, (b) replacing all unit tests forever with zero review, (c) compiling C. Answer: (a).
Short Answer: Why hash prompt versions? Answer: Reproducibility and attributing score changes to specific prompt text.
Short Answer: What is a regression gate? Answer: A threshold that blocks shipping when metrics fall vs. baseline.
Multiple Choice: Overfitting the golden set means: (a) prompts tuned only to eval cases, (b) perfect production forever, (c) free tokens. Answer: (a).
True/False: Module 13.2 begins with prompt structure craft after evaluation habits. Answer: True.

Key Takeaways

Evaluate prompts with fixed cases, controlled settings, and clear metrics.
Score format, task accuracy, and safety together.
Version prompts and gate releases on regressions.
Next module: 13.2 Prompt Structure.

Trainer’s Guide

Hands-on: Build a 20-case JSON extraction eval; compare two prompt variants with a pass-rate table.

Discussion: When is human labeling mandatory versus automatic checkers?

Recap: Prompt evaluation makes prompting scientific. Continue into Module 13.2 Prompt Structure.