Weight-space PEFT (LoRA, IA3) edits layers. Prefix tuning keeps every weight frozen and instead prepends trainable continuous “virtual tokens” to keys/values at each layer—soft task-specific prefixes in activation space. It pairs with the next lecture on lighter prompt tuning and Vol. 11’s PEFT overview.
Learning Objectives
By the end of this lesson, students should be able to:
- Define prefix tuning and its per-layer KV prefixes.
- Contrast prefix tuning with discrete prompt engineering.
- Compare depth of prefixes vs input-only prompt tuning.
- Configure PrefixTuningConfig in PEFT.
- Discuss reparameterization MLPs used during training.
- Place prefix tuning on the PEFT vs full FT spectrum.
Prefix tuning optimizes a small set of continuous prefix vectors that are prepended to the keys and values (and related states) in every Transformer layer. The pretrained weights stay frozen; task behavior is steered by these learned prefixes, often reparameterized through a small MLP for stable optimization.
Where the Prefix Lives
Discrete prompts
- Human-written text.
- Token IDs only.
- No gradient into soft vectors.
Prefix tuning
- Continuous vectors.
- Injected at every layer.
- Trained end-to-end.
Prompt tuning
- Soft tokens at input.
- Shallower than prefixes.
- Even fewer params.
| Property | Prefix tuning | LoRA |
|---|---|---|
| Frozen weights | Yes | Yes |
| Intervention | Activations / KV prefixes | Weight residuals |
| Params | Prefix length × layers × dim | r × fan-in/out × modules |
| Merge into W | Not naturally | Yes (merge_and_unload) |
PEFT Prefix Tuning
Virtual KV per layer
Attend with prefixes first
Update prefix params
Load another prefix pack
Strengths and Tradeoffs
Strengths
- No weight surgery; clean multi-task packs.
- Strong results on generation tasks historically.
- Orthogonal to quantization of the base.
Tradeoffs
- Uses sequence budget (virtual length).
- Cannot merge into dense W like LoRA.
- Less dominant than LoRA in modern LLM SFT.
“Prefix tuning is the same as writing a better system prompt.” System prompts are discrete tokens you type; prefixes are learned continuous parameters optimized with gradients—usually not human-readable text.
Knowledge Check
- Short Answer: Where are prefix vectors injected? Answer: As continuous prefixes to layer keys/values (per layer).
- True/False: Prefix tuning updates all Transformer weights. Answer: False—weights stay frozen.
- Multiple Choice: Versus prompt tuning, prefixes are: (a) deeper (per layer), (b) only CSS, (c) full FT. Answer: (a).
- Short Answer: Why use prefix_projection? Answer: MLP reparameterization stabilizes / improves optimization.
- True/False: Virtual tokens consume effective context length. Answer: True.
- Multiple Choice: LoRA merges into W more naturally than prefixes: (a) True statement, (b) False, (c) N/A. Answer: (a).
- Short Answer: Name the PEFT config class. Answer: PrefixTuningConfig.
- True/False: Discrete prompt engineering trains continuous prefixes by default. Answer: False.
- Multiple Choice: Prefix packs are useful for: (a) multi-task swapping, (b) only max-pooling, (c) DNS. Answer: (a).
- Short Answer: Which shallower soft-prompt method follows? Answer: Prompt tuning.
Key Takeaways
- Prefix tuning steers frozen models with per-layer continuous KV prefixes.
- Deeper and usually more expressive than input-only prompt tuning.
- Task packs swap without touching base weights.
- LoRA remains more common for large LLM SFT today.
- Next: Prompt Tuning.
Hands-on idea: Train prefix length 5 vs 30 on a small LM; plot trainable params vs validation loss.
Discussion prompt: Would you spend context tokens on virtual prefixes or keep them for RAG documents?
Recap: Prefix tuning learns deep soft prefixes while freezing the backbone. Continue with Prompt Tuning.