After instruction SFT, models still may be verbose, unsafe, or misaligned with what humans prefer. Reinforcement Learning from Human Feedback (RLHF) uses preference data to train a reward model, then optimizes the policy (usually with PPO) against that reward while staying close to the SFT reference. DPO later offers a simpler alternative.
Learning Objectives
By the end of this lesson, students should be able to:
- Describe the RLHF pipeline: SFT → RM → RL policy update.
- Explain preference pairs and reward model training.
- State why a KL penalty to the reference model is used.
- List practical costs and failure modes of PPO-based RLHF.
- Relate RLHF to the broader alignment goal.
- Preview why teams adopt DPO-style methods.
RLHF aligns a language model with human preferences by (1) collecting comparisons between model outputs, (2) fitting a reward model that scores responses, and (3) optimizing the LM policy with reinforcement learning to maximize reward while regularizing toward a reference policy (typically the SFT model).
The Classic Pipeline
Instruction-following base policy.
Humans (or AI) rank outputs.
Learn r(x, y) from pairs.
Maximize r with KL to SFT.
Why KL Regularization?
Without a penalty, RL can exploit reward-model quirks (“reward hacking”) and drift into gibberish that scores well. A KL term keeps the policy near the reference SFT distribution, preserving fluency and coverage.
| Stage | Data | Output artifact |
|---|---|---|
| SFT | Instruction–response | Chat-capable policy πSFT |
| RM | Preferred vs rejected y | Scalar reward model |
| PPO | Prompts + on-policy samples | Aligned policy πRL |
Strengths
- Optimizes directly for preference.
- Can improve helpfulness/harmlessness.
- Foundation of many chat products.
Tradeoffs
- Complex, unstable PPO training.
- Expensive human preference data.
- Reward hacking & sycophancy risks.
“RLHF teaches the model what is objectively true.” It teaches what raters prefer under a rubric. Preferences can encode biases, over-refusal, or style tastes—not ground truth.
Knowledge Check
- Short Answer: Name the three classic RLHF stages after pretraining. Answer: SFT, reward modeling, RL policy optimization (e.g., PPO).
- True/False: RLHF requires a reward model trained from preferences. Answer: True (in the classic pipeline).
- Multiple Choice: KL regularization mainly prevents: (a) tokenization, (b) excessive drift/reward hacking, (c) GPU existence, (d) batching. Answer: (b).
- Short Answer: What does a preference pair provide? Answer: A chosen vs rejected response for the same prompt.
- True/False: PPO-based RLHF is operationally simple and cheap. Answer: False—it is complex and costly.
- Multiple Choice: Reward hacking means: (a) perfect truthfulness, (b) exploiting RM flaws for high reward, (c) deleting the RM, (d) smaller context always. Answer: (b).
- Short Answer: What is the usual reference policy? Answer: The SFT model (or a fixed snapshot thereof).
- True/False: RLHF guarantees elimination of all hallucinations. Answer: False.
- Multiple Choice: A modern alternative to PPO RLHF is: (a) DPO, (b) only k-means, (c) only TF-IDF, (d) only dropout. Answer: (a).
- Short Answer: Why might RLHF increase sycophancy? Answer: Raters may reward agreeable answers over truthful ones.
Key Takeaways
- RLHF aligns LMs to preferences via RM + RL.
- KL to SFT stabilizes training and limits exploitation.
- It is powerful but heavy; DPO simplifies the stack.
- Next: DPO.
Whiteboard: Draw SFT → RM → PPO with KL arrow back to SFT.
Exercise: Have pairs write preference rubrics; discuss disagreements—alignment is social, not only math.
Recap: RLHF optimizes an LM against a learned reward from human preferences. Continue with DPO.