← Master Index
Vol. 06 Module 6.1 Lecture

ReLU

Neural Network Foundations

How This Lesson Fits Module 6.1

ReLU is the practical hidden-layer default students will use repeatedly in Volume 06. After seeing saturating activations like sigmoid and tanh, ReLU shows how a simple nonlinearity can improve gradient flow.

Learning Objectives

By the end of this lesson, students should be able to:

  • Define ReLU and its mathematical behavior.
  • Explain why ReLU trains deep feedforward networks well.
  • Describe sparse activations and the dying ReLU problem.
  • Implement ReLU with nn.ReLU and torch.relu.
  • Compare ReLU to sigmoid and tanh for hidden layers.
  • Choose when to consider LeakyReLU or GELU variants.
Definition

ReLU (Rectified Linear Unit) computes max(0, x): negative inputs become 0 and positive inputs pass through unchanged.

Simple Nonlinearity, Strong Gradient Flow

ReLU does not squash positive values into a narrow range. For positive inputs, the gradient is constant, which helps deep networks train faster than with saturating sigmoid or tanh. Negative inputs become zero, creating sparse activations. That sparsity is often useful, but a unit can become inactive for all examples if weights move badly, a behavior called dying ReLU.

ActivationRangeHidden-layer behavior
Sigmoid0 to 1Saturates; probability-friendly
Tanh-1 to 1Zero-centered but saturates
ReLU0 to infinityFast, sparse, non-saturating for positives
LeakyReLUSmall negative slopeReduces dying units
GELUSmooth probabilistic gateCommon in transformers

PyTorch Practice

ReLU can be used as a module inside nn.Sequential or as a functional tensor operation.

import torch from torch import nn x = torch.tensor([-2.0, -0.5, 0.0, 3.0]) print(torch.relu(x)) model = nn.Sequential( nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 2), ) logits = model(torch.randn(6, 10)) print(logits.shape)

Why ReLU Became the Default

Optimization

  • Useful gradient for positive activations
  • Computationally cheap
  • Works well with modern initialization

Representation

  • Creates sparse hidden features
  • Allows piecewise-linear functions
  • Stacks into complex boundaries

Cautions

  • Zero gradient for negative inputs
  • Unbounded positive outputs
  • May need normalization in deeper models

Strengths and Tradeoffs

Useful because

  • Fast and simple to compute.
  • Reduces saturation compared with sigmoid/tanh.
  • Effective default for many MLPs and CNNs.

Watch for

  • Negative side has zero gradient.
  • Dead units can stop learning.
  • Unbounded activations can grow without good initialization or normalization.

How It Flows

1. Affine

A layer computes weighted sums plus bias.

2. Rectify

ReLU clips negative values to zero.

3. Sparsify

Only active positive units pass signal onward.

4. Compose

Many ReLU layers build piecewise-linear functions.

Common Misconception

ReLU is not always best just because it is common. For transformer blocks, GELU is often used; for outputs, activation choice depends on the loss and target, not hidden-layer habit.

Knowledge Check

  1. Short Answer: What does ReLU compute? Answer: max(0, x).
  2. True/False: ReLU passes positive inputs unchanged. Answer: True.
  3. Multiple Choice: ReLU is commonly used in: (a) hidden layers, (b) target CSV files, (c) train/test splits only. Answer: (a).
  4. Short Answer: What is the dying ReLU problem? Answer: A unit outputs zero for all examples and receives no useful gradient.
  5. True/False: ReLU saturates for large positive inputs. Answer: False.
  6. Short Answer: Name one ReLU variant. Answer: LeakyReLU, ELU, or GELU.
  7. Multiple Choice: Negative ReLU input outputs: (a) 0, (b) -1 always, (c) probability. Answer: (a).
  8. Short Answer: Why can sparse activations help? Answer: They let only relevant units fire for an example.
  9. True/False: ReLU is differentiable at exactly zero in the classical sense. Answer: False; frameworks use a convention/subgradient.
  10. Short Answer: Why did ReLU help deep learning? Answer: Better gradient flow and simple computation.

Key Takeaways

  • ReLU is a simple, powerful hidden-layer activation.
  • It avoids positive-side saturation and supports sparse representations.
  • Dead units and unbounded activations are the main cautions.
  • Next, Softmax converts class logits into a probability distribution.
Trainer’s Guide

Hands-on idea: Show a batch of hidden activations before and after ReLU, then ask students to compute the fraction of zeros.

Discussion prompt: Why might sparsity be useful for representation learning?

Recap: ReLU keeps positive gradients alive while adding the nonlinearity hidden layers need. Continue with Softmax.