Tanh is sigmoid's zero-centered cousin. Studying it after sigmoid clarifies why activation range and gradient behavior matter before the module moves to ReLU.
Learning Objectives
By the end of this lesson, students should be able to:
- Define tanh and its output range.
- Compare tanh with sigmoid in hidden layers.
- Explain the benefit of zero-centered activations.
- Describe tanh saturation and vanishing gradients.
- Use
nn.Tanhin a small PyTorch model. - Identify settings where tanh remains useful.
The tanh activation maps real-valued inputs to the range -1 to 1, with tanh(0)=0 and saturation near both extremes.
Zero-Centered Squashing
Tanh keeps the smooth squashing behavior of sigmoid but centers outputs around zero. That can make optimization easier because positive and negative activations are balanced. Still, tanh saturates for large-magnitude inputs, so gradients can become tiny in deep stacks. It remains useful in recurrent networks, bounded control outputs, and educational comparisons.
| Property | Sigmoid | Tanh |
|---|---|---|
| Range | 0 to 1 | -1 to 1 |
| At zero | 0.5 | 0 |
| Zero-centered | No | Yes |
| Saturation | Both tails | Both tails |
| Common modern hidden default | Rare | Occasional; ReLU more common |
PyTorch Practice
Tanh can be inserted as a standard activation module, though ReLU is more common for basic feedforward networks.
Tanh in Practice
Advantages
- Zero-centered output
- Smooth derivative
- Bounded activations
Limitations
- Saturates at -1 and 1
- Can slow deep networks
- Often outperformed by ReLU variants
Use cases
- Recurrent gates/candidates
- Bounded output transformations
- Teaching activation tradeoffs
Strengths and Tradeoffs
Useful because
- Balances negative and positive activations.
- Works well when bounded hidden states are desired.
- Easy to interpret relative to sigmoid.
Watch for
- Vanishing gradients remain a serious issue in deep stacks.
- Computationally heavier than simple ReLU.
- May require careful initialization and normalization.
How It Flows
Layer computes an affine value.
Tanh compresses the value into -1 to 1.
Outputs around zero feed the next layer.
Large magnitudes produce tiny gradients near extremes.
Zero-centered does not mean gradient-safe. Tanh improves one sigmoid weakness but still saturates, so deep models can struggle if many tanh units sit near -1 or 1.
Knowledge Check
- Short Answer: What is the output range of tanh? Answer: -1 to 1.
- True/False: Tanh is zero-centered. Answer: True.
- Multiple Choice:
tanh(0)is: (a) 0, (b) 0.5, (c) 1. Answer: (a). - Short Answer: What problem does tanh share with sigmoid? Answer: Saturation and vanishing gradients.
- True/False: Tanh always prevents vanishing gradients. Answer: False.
- Short Answer: Why can zero-centered outputs help? Answer: They balance positive and negative signals for optimization.
- Multiple Choice: A common modern hidden default is: (a) ReLU, (b) manual HTML, (c) argmax. Answer: (a).
- Short Answer: Name one use case for tanh. Answer: Recurrent networks or bounded outputs.
- True/False: Tanh is differentiable. Answer: True.
- Short Answer: What happens for very large positive input? Answer: Output approaches 1.
Key Takeaways
- Tanh maps values to a zero-centered -1 to 1 range.
- It improves sigmoid's centering but still saturates.
- ReLU is usually the first hidden-layer default for feedforward models.
- Next, ReLU introduces the activation that made deeper networks easier to train.
Hands-on idea: Have students plot sigmoid, tanh, and ReLU on the same axes and compare where each function has useful gradient.
Discussion prompt: Why might bounded activations be desirable in some systems but harmful in others?
Recap: Tanh is a zero-centered squashing activation whose saturation explains why ReLU became so influential. Continue with ReLU.