Volume 07 focused on spatial structure. A CNN learns from neighboring pixels because images are arranged on a grid. Volume 08 asks a different question: what if the input is ordered in time or in a meaningful sequence, such as sensor readings, stock values, log events, music notes, or words in a sentence?
Recurrent Neural Networks (RNNs) are the first major answer. They introduce a running memory called the hidden state, allowing the model to combine the current input with what it has already seen. In real engineering systems, this idea appears in forecasting pipelines, anomaly detection, speech processing, and early NLP systems. This lesson gives the conceptual map for the entire module, including sequence data, time series, vanishing gradients, BPTT, LSTM, and GRU.
Learning Objectives
By the end of this lesson, students should be able to:
- Explain why ordered data requires a different modeling approach than dense networks or CNNs alone.
- Describe the recurrence idea: reuse the same cell across time while carrying a hidden state forward.
- Identify the input, hidden state, and output tensors in a simple RNN.
- Build a basic recurrent model in PyTorch using
nn.RNN. - Trace common RNN tensor shapes for batched sequence problems.
- Recognize why long sequences create optimization problems that motivate LSTM and GRU.
A Recurrent Neural Network is a neural network designed for ordered data. At each time step, it processes the current input together with a hidden state from the previous step, producing an updated hidden state and often an output.
Why Feedforward Thinking Breaks on Sequences
A feedforward network assumes every input feature is presented all at once. That works for a fixed-size tabular vector, and CNNs improve the situation for images by respecting spatial locality. But sequential data has an additional constraint: order matters. The sentence "model learns sequences" does not mean the same thing as "sequences learn model." Likewise, a temperature reading of 40 after a week of 39 is different from 40 after a week of 5.
If we flatten a sequence into one giant vector, we lose the clean notion of "what happened before" and "what happened now." We also force the model to use a fixed input length. RNNs solve this by applying the same transition rule repeatedly, one step at a time.
| Data type | Key structure | Natural model bias |
|---|---|---|
| Tabular records | No meaningful order | Dense layers |
| Images | 2D spatial locality | CNNs |
| Sequences | 1D order over time or position | RNNs and related sequence models |
The Core RNN Idea
An RNN cell is reused at every time step. The same parameters are shared from start to finish, but the hidden state changes as the sequence unfolds. This gives the model a compact, evolving summary of context.
Take the current token, feature vector, or measurement x_t.
Mix x_t with the previous hidden state h_(t-1).
Produce a new state h_t that carries forward context.
Optionally produce y_t for labeling, forecasting, or generation.
Conceptually, an RNN cell often looks like this:
h_t = f(W_x x_t + W_h h_(t-1) + b)
The important idea is not the exact equation. It is that h_t depends on both the current input and prior context.
How Recurrence Creates Memory
What stays the same
- The RNN cell architecture.
- The learned weights.
- The update rule at each step.
What changes over time
- The current input
x_t. - The hidden state
h_t. - The output at each position.
Why it matters
- Works on variable-length sequences.
- Reuses parameters efficiently.
- Models temporal dependence directly.
A Minimal PyTorch RNN
PyTorch provides a ready-made recurrent layer. The example below uses batch_first=True, so tensors are arranged as (batch, seq_len, input_size).
Two outputs are returned:
output: the hidden state at every time step.h_n: the final hidden state, often used as a summary of the whole sequence.
Reading the Shapes
| Tensor | Meaning | Shape here |
|---|---|---|
x | Batched input sequence | (4, 6, 3) |
output | Hidden state at each time step | (4, 6, 8) |
h_n | Final hidden state for each sample | (1, 4, 8) |
Where Basic RNNs Work Well
Simple RNNs are most useful when the dependency horizon is short to medium, or when the goal is instructional clarity. They appear in toy character models, small forecasting problems, compact classification systems, and as stepping stones toward more robust architectures.
Strengths
- Natural fit for ordered inputs.
- Parameter sharing across time.
- Simple mental model for sequence processing.
Limitations
- Struggles with long-range dependencies.
- Sequential computation reduces parallelism.
- Training can become unstable on long sequences.
Students often assume the hidden state is a perfect memory of everything that happened earlier. It is not. It is a learned, compressed summary with limited capacity. On long or difficult sequences, important information can fade or become distorted.
“RNNs are only for language.” In practice, they are useful for any ordered signal: clickstreams, ECG data, machine logs, weather measurements, control systems, and more. Language is just one especially important example.
Bridge from CNNs to RNNs
CNNs taught us to preserve structure instead of flattening everything immediately. RNNs apply the same engineering mindset to time: preserve order, reuse parameters, and let the model build hierarchy over a structured input. In Volume 09, this idea will connect naturally to tokenized text, embeddings, and sequence modeling for NLP.
Knowledge Check
- Short Answer: What problem does an RNN solve that a plain dense network handles poorly? Answer: It models ordered or temporal data while carrying context from earlier steps.
- True/False: An RNN uses different weights at every time step. Answer: False.
- Multiple Choice: The hidden state mainly represents: (a) the loss, (b) a running summary of context, (c) the optimizer state. Answer: (b).
- Short Answer: In PyTorch with
batch_first=True, what is the standard input layout fornn.RNN? Answer:(batch, sequence length, input size). - True/False: RNNs can process variable-length sequences. Answer: True.
- Multiple Choice: Which output often summarizes the full sequence? (a) the first input, (b) the final hidden state, (c) the optimizer gradients. Answer: (b).
- Short Answer: Why is order important in sequence modeling? Answer: Because changing the order changes the meaning or dynamics of the data.
- True/False: A simple RNN is usually better than LSTM or GRU at preserving very long dependencies. Answer: False.
- Multiple Choice: Which earlier volume provides the spatial-modeling bridge into this topic? (a) Vol. 05, (b) Vol. 07, (c) Vol. 10. Answer: (b).
- Short Answer: What module topics are motivated by basic RNN limitations? Answer: Vanishing gradients, exploding gradients, BPTT, LSTM, and GRU.
Key Takeaways
- RNNs are designed for sequence structure in the same spirit that CNNs are designed for spatial structure.
- The hidden state allows the model to carry context from earlier time steps.
- PyTorch
nn.RNNreturns both per-step outputs and a final hidden summary. - Simple RNNs introduce the core sequence idea, but they struggle on long sequences.
- Next, Sequence Data examines the kinds of inputs that make recurrence necessary.
Hands-on idea: Give learners short sequences such as daily temperatures or token IDs and have them label what information the hidden state should ideally preserve.
Discussion prompt: Ask students to compare an image classification pipeline with a sentiment-analysis pipeline and identify why spatial order and temporal order demand different inductive biases.
Recap: RNNs extend neural networks from static inputs to ordered streams by reusing one cell over time and carrying a hidden state forward. Continue with Sequence Data.