Full Fine Tuning — Module 12.4

How This Lesson Fits the Module & Volume

Volume 11 surveyed fine-tuning and PEFT at a high level. Module 12.4 goes deep: we start with full fine-tuning—updating every trainable weight—as the baseline every PEFT method must beat on cost, memory, and multi-adapter logistics.

After serving optimizations in 12.3, this module asks how to specialize models. Next lectures cover SFT, then LoRA-family and prompt/adapter methods.

Learning Objectives

By the end of this lesson, students should be able to:

Define full fine-tuning versus freezing subsets of layers.
Estimate optimizer-state memory (Adam) relative to parameter count.
List risks: catastrophic forgetting, checkpoint size, multi-task sprawl.
Launch a minimal full-FT training step with Hugging Face Transformers.
Decide when full FT is justified over PEFT.
Connect full FT to later PEFT comparisons in this module.

Definition

Full fine-tuning continues training a pretrained model by updating all (or essentially all) parameters on a downstream dataset. Gradients and optimizer states are maintained for every weight, producing a complete new checkpoint the size of the base model.

Cost Stack

Resource	Full FT impact	PEFT contrast
Trainable params	100% of model	~0.1–1% typical (LoRA)
Adam states	~2× params (m, v) extra	Only on tiny adapters
Checkpoint	Full model copy per run	Small adapter file
Multi-task serving	One model per task (heavy)	Swap adapters / merge
Forgetting risk	Higher if data narrow	Often lower (frozen base)

When Full FT Still Wins

Large domain shift

New language / modality quirks.
Adapters may underfit.

Abundant compute

Multi-GPU / long budgets.
One canonical specialist model.

Research baselines

Upper-bound quality study.
Compare PEFT gaps fairly.

Minimal Hugging Face Loop

from transformers import AutoModelForCausalLM, AutoTokenizer, Trainer, TrainingArguments

model_id = "meta-llama/Llama-3.2-1B"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)  # all params trainable

# Ensure every parameter requires grad (full FT)
for p in model.parameters():
    p.requires_grad = True

args = TrainingArguments(
    output_dir="ft-full",
    per_device_train_batch_size=1,
    gradient_accumulation_steps=8,
    learning_rate=2e-5,
    num_train_epochs=1,
    fp16=True,
    logging_steps=10,
)
# Trainer(model=model, args=args, train_dataset=..., tokenizer=tok).train()
print(sum(p.numel() for p in model.parameters() if p.requires_grad))

1. Load base

Pretrained checkpoint

2. Unfreeze all

Optimizer on every W

3. Train

Task / SFT data

4. Save full

New multi-GB weights

Strengths and Tradeoffs

Strengths

Maximum capacity to fit the task.
No adapter hyperparameter surface.
Simple mental model.

Tradeoffs

Expensive memory and storage.
Hard to maintain many variants.
Easier to overwrite pretrained skills.

Common Misconception

“Full fine-tuning always beats LoRA on quality.” On many instruction datasets, well-tuned LoRA/QLoRA matches full FT within noise—while costing far less. Full FT is a capacity ceiling, not an automatic win.

Knowledge Check

Short Answer: What fraction of weights does full FT update? Answer: Essentially all trainable parameters.
True/False: Adam full FT needs optimizer states for every parameter. Answer: True.
Multiple Choice: Full FT checkpoints are typically: (a) tiny adapter files, (b) full model-sized, (c) CSS-only. Answer: (b).
Short Answer: Name one risk of aggressive full FT on narrow data. Answer: Catastrophic forgetting (or overfitting).
True/False: Serving 20 full-FT task models is lighter than 20 LoRA adapters. Answer: False.
Multiple Choice: PEFT usually trains: (a) a small parameter subset, (b) only the tokenizer, (c) nothing. Answer: (a).
Short Answer: Give one case where full FT is justified. Answer: Large domain shift / ample compute / research ceiling (any).
True/False: Module 11 already introduced PEFT; 12.4 deepens methods. Answer: True.
Multiple Choice: Next lecture focuses on: (a) SFT, (b) CNNs, (c) DNS. Answer: (a).
Short Answer: What memory term often dominates full FT GPUs? Answer: Optimizer states (and activations/gradients).

Key Takeaways

Full FT updates the entire model—maximum flexibility, maximum cost.
Optimizer states and checkpoint sprawl drive PEFT adoption.
Use full FT when capacity or domain shift demands it.
This module’s PEFT methods are cheaper alternatives for most LLM SFT.
Next: SFT (Supervised Fine-Tuning).

Trainer’s Guide

Hands-on idea: Compare disk size and trainable param counts for full FT vs LoRA r=16 on a 1B model.

Discussion prompt: For ten enterprise tenants, would you full-FT ten models or ship ten adapters?

Recap: Full fine-tuning is the costly baseline that PEFT methods approximate. Continue with SFT.