SFT, RLHF, and DPO are techniques. Alignment is the goal: making model behavior match human intentions, values, and constraints in deployment. This lecture connects methods to product policies, evals, and limits—including why hallucinations persist.
Learning Objectives
By the end of this lesson, students should be able to:
- Define alignment in the LLM product sense (helpful, honest, harmless).
- Map technical methods (SFT, preferences, constitutions, filters) to alignment.
- Distinguish specification, training, and oversight problems.
- Design a minimal alignment eval suite for a use case.
- Explain residual risks after post-training.
- Relate open vs closed model policies to alignment controls.
Alignment (for deployed LLMs) means steering a model so its outputs and tool actions reliably advance the user’s intended goals within safety, legal, and organizational constraints—not merely maximizing next-token likelihood on raw internet text.
Three Layers of the Problem
Specification
- What should “good” mean?
- Policies, rubrics, red lines.
Training
- SFT / RLHF / DPO / constitutions.
- Data that encodes the spec.
Oversight
- Evals, monitors, human review.
- Runtime filters and permissions.
Helpful, Honest, Harmless
| Axis | Desired behavior | Typical failure |
|---|---|---|
| Helpful | Useful, on-task answers | Refusal theater / laziness |
| Honest | Calibrated, grounded claims | Hallucination, overconfidence |
| Harmless | Respects safety policy | Jailbreaks, dual-use help |
Write allowed/disallowed behaviors.
Encode policy in weights.
Prompt, tools, permissions.
Eval + incident response.
What Post-Training Buys
- Better default assistant behavior.
- Reduced obvious unsafe completions.
- More consistent tone and format.
What Remains
- Jailbreaks and distribution shift.
- Conflicting user vs org goals.
- Grounding gaps without RAG.
“Once a model is aligned, it stays aligned forever.” New tools, longer contexts, fine-tunes, and novel attacks change behavior. Alignment is a continuous process of training, evaluation, and system design.
Knowledge Check
- Short Answer: What is alignment trying to achieve for LLMs? Answer: Behavior that matches intended goals and constraints, not raw pretraining priors alone.
- True/False: Alignment is only a math loss, never a product policy problem. Answer: False—specification and oversight are essential.
- Multiple Choice: HHH commonly expands to: (a) heavy, hot, huge, (b) helpful, honest, harmless, (c) hash, heap, hook, (d) hidden, hard, hasty. Answer: (b).
- Short Answer: Name one training method used for alignment. Answer: RLHF, DPO, instruction SFT, constitutional methods, etc.
- True/False: Runtime filters and tool permissions are part of alignment practice. Answer: True.
- Multiple Choice: A specification failure means: (a) GPUs melted, (b) the written goals/policies were wrong or vague, (c) softmax vanished, (d) tokens became images. Answer: (b).
- Short Answer: Why do hallucinations survive “aligned” chat models? Answer: Preference tuning does not install a perfect truth database; grounding is still needed.
- True/False: Fine-tuning an aligned model on arbitrary data can undo safety behavior. Answer: True—regression is common without constraints/evals.
- Multiple Choice: Continuous alignment includes: (a) never evaluating, (b) monitoring and updating policies/evals, (c) deleting logs always, (d) only marketing claims. Answer: (b).
- Short Answer: List the three problem layers named in this lecture. Answer: Specification, training, and oversight.
Key Takeaways
- Alignment is the goal; SFT/RLHF/DPO are tools.
- Specify, train, and oversee—none alone is enough.
- Helpful / honest / harmless trade off in real products.
- Next: Hallucination examines a core honesty failure.
Policy workshop: Draft a one-page acceptable-use policy for a student tutor bot; mark what belongs in SFT vs system prompt vs filters.
Red team: Attempt jailbreaks ethically in a sandbox; log which defenses catch them.
Recap: Alignment spans policy, training, and oversight around LLM behavior. Continue with Hallucination.