← Master Index
Vol. 08 Module 8.1 Lecture

Exploding Gradient

Recurrent Networks

How This Lesson Fits the Module & Engineering Practice

Vanishing gradients explain why recurrent networks can forget. Exploding gradients explain why they can become numerically unstable. Both arise because training an RNN means backpropagating through a long chain of repeated transformations.

In engineering practice, exploding gradients show up as wildly unstable losses, NaN parameters, or sudden training divergence. This is why gradient clipping is so common in recurrent training code, and why lectures on BPTT, LSTM, and GRU matter operationally, not just theoretically.

Learning Objectives

By the end of this lesson, students should be able to:

  • Define the exploding gradient problem in recurrent training.
  • Explain why repeated multiplication can amplify gradients instead of shrinking them.
  • Recognize practical warning signs such as unstable loss spikes and NaN values.
  • Use gradient clipping in PyTorch to stabilize training.
  • Compare exploding gradients with vanishing gradients conceptually and operationally.
  • Understand why stability controls are standard in RNN, LSTM, and GRU workflows.
Definition

Exploding gradient is the phenomenon where gradients grow extremely large during backpropagation, often causing unstable updates, numerical overflow, or training divergence.

Why Gradients Blow Up

Backpropagation through an unrolled RNN repeatedly multiplies derivatives across time steps. If the effective magnitudes in that chain are greater than 1 on average, the gradient can grow exponentially. Instead of learning signal becoming too weak, it becomes too strong and erratic.

Large gradients mean large parameter updates. A single bad step can throw the model far away from a useful region of parameter space, especially when combined with a high learning rate.

IssueGradient scaleObserved behavior
VanishingTinyNo credit reaches early steps
ExplodingHugeUpdates become unstable or divergent

Operational Symptoms

During training

  • Loss spikes suddenly.
  • Metrics jump unpredictably.
  • Training diverges after seeming fine.

In tensors

  • Very large gradient norms.
  • inf or NaN values.
  • Weights change too aggressively.

In deployment quality

  • Training becomes unreliable.
  • Model checkpoints vary wildly.
  • Reproducibility suffers.

Gradient Clipping: The Standard Defense

The most common practical response is gradient clipping. Rather than letting the full gradient norm grow without bound, we cap it before the optimizer step. This does not fix every underlying issue, but it prevents catastrophic updates.

1. Forward pass

Compute predictions and loss.

2. Backward pass

Accumulate gradients on parameters.

3. Clip

Limit gradient norm to a safe threshold.

4. Step

Run the optimizer update.

PyTorch Example: Clip Before Update

import torch from torch import nn model = nn.RNN(input_size=10, hidden_size=32, batch_first=True) head = nn.Linear(32, 4) optimizer = torch.optim.Adam(list(model.parameters()) + list(head.parameters()), lr=1e-3) x = torch.randn(16, 40, 10) y = torch.randint(0, 4, (16,)) output, h_n = model(x) logits = head(output[:, -1, :]) loss = nn.CrossEntropyLoss()(logits, y) optimizer.zero_grad() loss.backward() torch.nn.utils.clip_grad_norm_(list(model.parameters()) + list(head.parameters()), max_norm=1.0) optimizer.step()

The clipping threshold is task-dependent. Common values include 0.5, 1.0, or 5.0, but engineers usually tune the threshold by observing training behavior and gradient norms.

Exploding Gradient Versus High Learning Rate

These problems are related but not identical. A learning rate that is too high can destabilize training even when gradients are reasonable. Exploding gradients, by contrast, mean the backward pass itself produced excessively large updates before the optimizer scaling decision.

Stability Tools

  • Gradient clipping.
  • Smaller learning rates.
  • Careful initialization and normalization.

What to Monitor

  • Gradient norms over time.
  • Loss curves for spikes.
  • Numerical warnings or NaNs.
Common Mistake

Some teams add clipping and stop investigating. Clipping is a stabilizer, not a substitute for understanding. If sequence length, initialization, data scaling, or learning rate are badly chosen, clipping may hide the symptoms without fixing the modeling issue.

Misconception

“Exploding gradients mean the model is learning faster.” Large gradients do not imply better learning. Beyond a point, they destroy useful optimization signal by pushing parameters into unstable regions.

Why This Matters for BPTT

Exploding gradients occur during Backpropagation Through Time, the training procedure that unrolls recurrence across the sequence. Once you understand both vanishing and exploding behavior, BPTT becomes much easier to reason about in implementation.

These stability concerns also explain why sequence architectures evolved beyond simple RNNs toward gated recurrent models and, later in the curriculum, more advanced NLP architectures.

Knowledge Check

  1. Short Answer: What is an exploding gradient? Answer: A gradient that becomes extremely large during backpropagation, causing unstable updates.
  2. True/False: Exploding gradients can produce NaN values during training. Answer: True.
  3. Multiple Choice: The most common defense is: (a) max pooling, (b) gradient clipping, (c) image augmentation. Answer: (b).
  4. Short Answer: Why are exploding gradients especially relevant in RNNs? Answer: Because gradients are repeatedly multiplied across many time steps in the unrolled recurrent chain.
  5. True/False: Exploding gradients and high learning rate are exactly the same issue. Answer: False.
  6. Multiple Choice: Which PyTorch utility is commonly used here? (a) clip_grad_norm_, (b) flatten, (c) permute_pixels. Answer: (a).
  7. Short Answer: What can happen if the optimizer steps on unclipped huge gradients? Answer: Parameters can jump to unstable regions, causing divergence or overflow.
  8. True/False: Clipping guarantees a model will learn long-range dependencies. Answer: False.
  9. Multiple Choice: Which next lecture explains the actual training procedure where these gradients are propagated through sequence time steps? (a) BPTT, (b) Hidden State, (c) Convolution. Answer: (a).
  10. Short Answer: What metric should engineers often inspect when diagnosing exploding gradients? Answer: Gradient norms over time.

Key Takeaways

  • Exploding gradients are the large-gradient counterpart to vanishing gradients in recurrent training.
  • They can destabilize optimization, cause loss spikes, and create numerical failures.
  • Gradient clipping is a standard practical safeguard in PyTorch recurrent workflows.
  • Stability still depends on good learning rates, scaling, and sequence design.
  • Next, BPTT (Backprop Through Time) ties these ideas into the full training process.
Trainer’s Guide

Hands-on idea: Run one toy recurrent training loop with clipping disabled and another with clipping enabled, then compare the loss curves and gradient norms.

Discussion prompt: Ask students whether clipping should be considered a patch, a best practice, or both.

Recap: Exploding gradients make recurrent optimization unstable, which is why gradient clipping and careful training control are standard practice. Continue with BPTT.