Self-consistency improves answers by voting across samples. Reflection improves answers by critique: generate a draft, examine it against criteria, then revise. It is sequential and often cheaper than large-n sampling when a single thoughtful repair fixes the issue.
Reflection connects forward to prompt chaining (draft → critique → revise as stages) and to guardrails (policy checks as structured critique). It also complements role prompting when you cast separate generator and reviewer personas.
Learning Objectives
By the end of this lesson, students should be able to:
- Define reflection (self-refine) prompting as draft → critique → revise.
- Write critique rubrics that catch concrete failure modes.
- Separate generator and reviewer prompts to reduce rubber-stamping.
- Decide between reflection, self-consistency, and ToT for a workload.
- Limit revision loops to avoid endless polish without quality gains.
- Log critiques for debugging and evaluation.
Reflection (also called self-refine / self-critique) is a prompting pattern where the model produces an initial output, then generates explicit feedback against a checklist or rubric, and finally produces an improved revision that addresses that feedback.
The Three Passes
Solve the user task
Score vs rubric
Fix cited issues
Draft Prompt
Critique Prompt
Revise Prompt
Reflection vs Related Patterns
| Pattern | Mechanism | Best when |
|---|---|---|
| Reflection | Critique + revise one lineage | Writable rubrics; fixable drafts |
| Self-consistency | Vote across samples | Discrete answers; diversity helps |
| ToT | Branching search | Hard planning / puzzles |
| CoT alone | One reasoned pass | Cheap multi-step baseline |
Strengths and Tradeoffs
Strengths
- Targets known failure modes via rubrics.
- Often 2–3 calls beat large-n voting on prose.
- Critiques create an audit trail.
Tradeoffs
- Weak rubrics yield empty praise.
- Models may “fix” issues without real change.
- Extra latency versus single-shot.
“Ask the model ‘are you sure?’ and that is reflection.” Vague self-checks produce vague confidence. Effective reflection uses a specific rubric, requires cited issues, and runs a dedicated revise step that must address those issues.
Knowledge Check
- Short Answer: What are the three stages of reflection? Answer: Draft, critique, revise.
- True/False: Reflection is the same as majority voting. Answer: False.
- Multiple Choice: A good critique prompt includes: (a) a concrete rubric, (b) only “be better,” (c) CUDA flags. Answer: (a).
- Short Answer: Why separate reviewer and writer roles? Answer: Reduces rubber-stamping; clearer critique focus.
- True/False: Endless revision loops always improve quality. Answer: False—gains diminish; cap iterations.
- Multiple Choice: Reflection often beats large-n voting for: (a) long prose with rubrics, (b) coin flips, (c) kernel convolution. Answer: (a).
- Short Answer: Name one failure mode of reflection. Answer: Empty praise / fake fixes / weak rubrics (any).
- True/False: Critiques can be logged for evaluation. Answer: True.
- Multiple Choice: Closest orchestration cousin: (a) prompt chaining, (b) max pooling, (c) byte-pair encoding. Answer: (a).
- Short Answer: What should the revise step output? Answer: An improved artifact that addresses cited issues (not a meta-discussion).
Key Takeaways
- Reflection = draft, rubric-based critique, then revise.
- Specific checklists beat vague “are you sure?” prompts.
- Cap loops; verify that revisions actually change the output.
- Next: Role Prompting.
Hands-on idea: Same draft with a vague critique vs a four-point rubric; compare revision quality.
Discussion prompt: When should critique be a separate model/role versus the same model in one multi-turn thread?
Recap: Reflection improves outputs through explicit critique and revision. Continue with Role Prompting.