Instruction Tuning — Module 11.4

How This Lesson Fits the Module & Volume

A pretrained LLM completes text; it does not naturally behave like a helpful assistant. Instruction tuning (supervised fine-tuning on instruction–response pairs) teaches the model to follow natural-language intents. It is the bridge from base models to chat models, and usually precedes RLHF / DPO.

Learning Objectives

By the end of this lesson, students should be able to:

Define instruction tuning as SFT on (instruction, response) data.
Contrast base completion models with instruction-tuned chat models.
Describe data mix quality: diversity, clarity, refusal examples.
Sketch a Hugging Face SFT training loop at a high level.
Explain how chat templates format multi-turn dialogues.
Relate instruction tuning to later preference alignment stages.

Definition

Instruction tuning is supervised fine-tuning of a pretrained language model on a curated dataset of instructions (and often multi-turn dialogues) paired with high-quality target responses, so the model learns to follow user intents rather than merely continue text.

Base vs Instruction-Tuned

Property	Base LM	Instruction-tuned LM
Objective at train time	Next token on raw text	Next token on instruction–response text
Default behavior	Continues the prompt	Answers / follows the ask
UI fit	Completion APIs	Chat / assistant APIs
Still needs?	Heavy prompt craft	Often preference tuning + safety

Data Matters More Than Magic

Good Mix

Diverse tasks and domains.
Clear instructions, verified answers.
Multi-turn and tool-use examples.

Failure Modes

Homogeneous synthetic spam.
Noisy or contradictory labels.
Missing refusal / safety cases.

Minimal HF-Style SFT Sketch

# Conceptual sketch — use official Trainer/TRL APIs in production
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model

model_id = "meta-llama/Llama-3.1-8B"  # example base
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")

# Format each row with the model's chat template, then causal LM loss
# on response tokens only (prompt tokens masked).
peft_cfg = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"])
model = get_peft_model(model, peft_cfg)
# train with SFTTrainer / custom loop on instruction dataset...

Collect

Instructions + gold responses.

Format

Chat template / special tokens.

SFT

Causal LM loss on answers.

Eval

Follow-rate, quality, safety.

Common Misconception

“Instruction tuning creates new world knowledge.” It mainly reshapes behavior (how to respond). Factual coverage still mostly comes from pretraining and from context you provide at inference (RAG/tools).

Knowledge Check

Short Answer: What data format does instruction tuning use? Answer: Instruction (and dialogue) paired with target responses.
True/False: A base LM already behaves like a polished chat assistant by default. Answer: False—it tends to continue text unless tuned.
Multiple Choice: Instruction tuning is primarily: (a) unsupervised clustering, (b) supervised fine-tuning, (c) k-NN retrieval, (d) PCA. Answer: (b).
Short Answer: What is a chat template? Answer: A model-specific formatting of roles/turns into the token sequence the model expects.
True/False: Preference methods like RLHF often come after SFT. Answer: True.
Multiple Choice: Poor instruction data often causes: (a) perfect reasoning, (b) brittle or sycophantic behavior, (c) free GPUs, (d) zero loss always. Answer: (b).
Short Answer: Should loss be computed on prompt tokens? Answer: Usually no—mask prompts; supervise response tokens.
True/False: Instruction tuning alone guarantees alignment with human values. Answer: False—it is necessary but not sufficient; preference/safety stages help.
Multiple Choice: A common efficient SFT approach uses: (a) only full 70B updates always, (b) LoRA/PEFT adapters, (c) deleting the tokenizer, (d) random labels. Answer: (b).
Short Answer: Name one thing instruction tuning does NOT primarily add. Answer: Brand-new factual knowledge beyond what pretraining/context provide.

Key Takeaways

Instruction tuning = SFT that teaches following intents.
Data quality and chat formatting dominate outcomes.
It prepares models for preference alignment and products.
Next: Fine-Tuning generalizes adaptation beyond instructions.

Trainer’s Guide

Lab: Write 20 high-quality instruction pairs for a domain (e.g., SQL help). Compare answers from base vs instruct checkpoints if available.

Prompt: Why might synthetic instruction data both help and hurt?

Recap: Instruction tuning turns completers into followers of user intents. Continue with Fine-Tuning.