SFT (Supervised Fine-Tuning) — Module 12.4

How This Lesson Fits the Module & Volume

Full fine-tuning describes how many weights you update. Supervised Fine-Tuning (SFT) describes what objective you optimize: next-token loss on curated instruction–response pairs. Volume 11 covered instruction tuning; here we operationalize SFT data, masking, and trainers used before PEFT methods like LoRA.

Learning Objectives

By the end of this lesson, students should be able to:

Define SFT and its place before RLHF/DPO in alignment stacks.
Format chat/instruction examples and apply response-only loss masks.
Run SFT with Hugging Face TRL SFTTrainer (conceptually).
List data quality failure modes (leakage, style collapse, toxic demos).
Explain that SFT can be full FT or PEFT under the hood.
Relate SFT checkpoints to later preference optimization (Vol. 11.4).

Definition

Supervised Fine-Tuning (SFT) trains a language model on labeled prompt–completion examples with standard cross-entropy (teacher forcing). The model learns to imitate high-quality demonstrations—often the first stage of aligning base models into assistants—whether you update all weights or only PEFT adapters.

Alignment Pipeline Context

1. Base LM

Pretrain on web-scale text

2. SFT

Imitate instructions

3. Preference

RLHF / DPO (Vol. 11)

4. Serve

Eval + guardrails

Concern	SFT practice
Loss mask	Often train on assistant tokens only
Format	Chat templates (roles, special tokens)
Data scale	Thousands to millions of demos
Packing	Concatenate short examples for efficiency
PEFT hook	Same SFT loss on LoRA/QLoRA params

Response-Only Loss

If you backprop through the user prompt tokens, the model wastes capacity memorizing prompts. Mask those positions (label = -100 in HF) so gradients come from the assistant span.

from datasets import Dataset
from trl import SFTTrainer, SFTConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig

model_id = "meta-llama/Llama-3.2-1B-Instruct"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

rows = [
    {"messages": [
        {"role": "user", "content": "Summarize KV caching in one sentence."},
        {"role": "assistant", "content": "KV caching stores past keys/values so decode is incremental."},
    ]}
]
ds = Dataset.from_list(rows)

peft_cfg = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"])
args = SFTConfig(output_dir="sft-out", max_seq_length=512, per_device_train_batch_size=1)

# trainer = SFTTrainer(model=model, args=args, train_dataset=ds, peft_config=peft_cfg,
#                      processing_class=tok)
# trainer.train()
print("SFT = demonstration CE loss; PEFT optional")

Data Quality Checklist

Good demos

Clear instructions.
Consistent style/format.
Cover edge cases.

Bad demos

Contradictory labels.
Prompt leakage into answers.
Toxic or unsafe targets.

Eval

Held-out instruction sets.
Rubrics / LLM-as-judge.
Regression on base skills.

Strengths and Tradeoffs

Strengths

Teaches format and task following quickly.
Simple CE objective, mature tooling.
Works with full FT or PEFT.

Tradeoffs

Imitation ≠ preference ranking.
Garbage demos → garbage policy.
Can overfit verbose templates.

Common Misconception

“SFT means LoRA.” SFT is the training objective and data recipe. LoRA/QLoRA/full FT are parameterization choices for how you implement that recipe.

Knowledge Check

Short Answer: What loss does SFT typically use? Answer: Next-token cross-entropy on demonstrations.
True/False: SFT must update all model weights. Answer: False—PEFT SFT is common.
Multiple Choice: Response-only masking skips loss on: (a) assistant tokens, (b) prompt/user tokens, (c) EOS forever. Answer: (b).
Short Answer: Where does SFT sit relative to RLHF/DPO? Answer: Usually before preference optimization.
True/False: Chat templates matter for multi-turn SFT. Answer: True.
Multiple Choice: TRL’s common SFT helper is: (a) SFTTrainer, (b) MaxPool2d, (c) BeamSearchCSS. Answer: (a).
Short Answer: Name one data failure mode. Answer: Contradictory labels / toxicity / leakage (any).
True/False: SFT alone always encodes human preference rankings. Answer: False—it imitates demos.
Multiple Choice: Vol. 11 related lecture: (a) instruction tuning, (b) pooling, (c) DNS. Answer: (a).
Short Answer: Which PEFT method is next? Answer: LoRA.

Key Takeaways

SFT = supervised imitation on instruction data.
Mask prompts; train on assistant completions.
Orthogonal to full FT vs PEFT parameterization.
Data quality dominates SFT success.
Next: LoRA.

Trainer’s Guide

Hands-on idea: Build 50 chat examples, train tiny SFT with/without prompt masking; compare prompt echoing.

Discussion prompt: When is a bigger dirty dataset worse than a smaller clean one for SFT?

Recap: SFT teaches models to follow demonstrations; PEFT methods change what you store and train. Continue with LoRA.