Gradient Clipping — Module 6.2

How This Lesson Fits the Module

Recurrent networks, transformers, and deep MLPs can produce exploding gradients—weight updates so large that loss becomes NaN. Gradient clipping caps gradient norms or values after backward() and before optimizer.step(), keeping training stable without changing the architecture.

Clipping is cheap insurance in RNN/LSTM/GRU and large-language-model training loops.

Prior Lesson Warmup eases early steps. Clipping bounds worst-case gradient spikes throughout training.

Learning Objectives

By the end of this lesson, students should be able to:

Apply clip_grad_norm_ and clip_grad_value_ correctly in the training loop.
Explain global norm clipping vs per-value clipping.
Place clipping after loss.backward() and before optimizer.step().
Choose max_norm values (typical 0.5–5.0 for RNNs, 1.0 common default).
Monitor gradient norms to detect instability before NaNs appear.
Combine clipping with mixed precision (unscale gradients first).

Two Clipping Strategies

Function	Mechanism	When to use
`clip_grad_norm_(params, max_norm)`	Scales all gradients if total L2 norm exceeds `max_norm`	Default for RNNs, transformers
`clip_grad_value_(params, clip_value)`	Clamps each gradient element to [−clip, +clip]	Outlier-heavy gradients

Norm Clipping in the Training Loop

Compute gradients, clip globally, then step. Log the pre-clip norm occasionally to tune max_norm.

max_norm = 1.0

for inputs, targets in train_loader:
    optimizer.zero_grad()
    outputs = model(inputs.to(device))
    loss = criterion(outputs, targets.to(device))
    loss.backward()

    # Returns total norm before clipping (useful for logging)
    grad_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm)

    optimizer.step()

    if batch_idx % 100 == 0:
        print(f"grad_norm={grad_norm:.2f}")

Value Clipping Alternative

torch.nn.utils.clip_grad_value_(model.parameters(), clip_value=0.5)

Critical Mistake — Clipping Before backward()

Gradients do not exist until loss.backward() completes. Clipping before backward is a no-op or error. Order: backward → clip → step.

Clipping with Mixed Precision

Under GradScaler, gradients are scaled. Unscale before clipping so max_norm applies to real gradient magnitudes.

scaler.scale(loss).backward()
scaler.unscale_(optimizer)
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm)
scaler.step(optimizer)
scaler.update()

Exploding vs Vanishing Gradients

Clipping fixes explosions (norm >> 1, NaN loss). Vanishing gradients (norm ≈ 0, no learning) need architectural fixes—residual connections, better init, or different activation—not clipping alone.

Engineering Habit — Alert on grad_norm

Log a histogram of gradient norms. Sudden 100× spikes often precede NaN by a few batches—clip and investigate data bugs.

Knowledge Check

Short Answer: Where does clipping go in the loop? Answer: After backward, before optimizer step.
True/False: clip_grad_norm_ clips each weight independently. Answer: False—scales all gradients together to cap global norm.
Multiple Choice: NaN loss in RNN training — try: (a) clip_grad_norm_, (b) remove backward, (c) clip before forward. Answer: (a).
Short Answer: What does max_norm=1.0 mean? Answer: Total L2 norm of gradients capped at 1.0.
Short Answer: Why unscale before clip with AMP? Answer: Gradients are scaled up; norm must reflect true values.
True/False: Clipping solves vanishing gradients. Answer: False—addresses exploding gradients.
Multiple Choice: clip_grad_value_(..., 0.5) clamps: (a) loss, (b) each grad element, (c) weights. Answer: (b).
Short Answer: What return value helps monitoring? Answer: Pre-clip total norm from clip_grad_norm_.
Short Answer: Typical max_norm for transformers? Answer: Often 1.0 (task-dependent).
Multiple Choice: Gradient accumulation + clipping: clip (a) each micro-batch, (b) after accumulated backward, (c) never. Answer: (b) before step.

Key Takeaways

Clip after backward: clip_grad_norm_ is the default for sequence models.
With AMP, unscale_ before clipping.
Log gradient norms; clipping prevents explosions, not vanishing.
Next: Mixed Precision—train faster with float16.

Trainer’s Guide

Hands-on idea: Train an RNN without clipping until NaN; restart with max_norm=1 and compare.

Discussion prompt: Does clipping change the loss landscape or just step size?

What’s Next Speed up training with Mixed Precision while keeping clipping in the scaled loop.