RNNs are only meaningful if they can carry context. The hidden state is the mechanism that makes recurrence more than repeated feedforward computation. It is the internal memory passed from one step to the next.
In engineering practice, understanding hidden state explains why a recurrent model can detect a trend, remember a trigger event, or summarize a sentence. It also explains why recurrence sometimes fails: the state is compressed, finite, and hard to optimize over long horizons. This lesson therefore sits at the center of the module.
Learning Objectives
By the end of this lesson, students should be able to:
- Define the hidden state as the internal memory of a recurrent model.
- Explain how hidden states are updated from the current input and previous state.
- Interpret PyTorch outputs such as
outputandh_nin recurrent layers. - Distinguish between per-step representations and final-sequence summaries.
- Describe how hidden state capacity limits what an RNN can remember.
- Connect hidden-state behavior to downstream tasks such as forecasting and classification.
The hidden state is the learned internal representation that a recurrent network carries from one time step to the next. It acts as a compressed summary of relevant past information.
Why Memory Must Be Internal
Suppose a model sees a stream of characters and needs to decide whether the next symbol completes a known pattern. The current character alone is not enough. The model must know what came before. Rather than storing the full history explicitly, an RNN stores a learned summary in the hidden state.
This design is elegant because it reuses a fixed-size vector regardless of sequence length. It is also difficult because a fixed-size vector may not preserve everything the task needs.
The Update Rule
At time step t, the hidden state is updated using both the current input and the previous hidden state:
h_t = f(x_t, h_(t-1))
In a simple RNN, this often becomes a weighted sum followed by a nonlinearity. The exact math matters less than the flow of information: some part of the past is kept, some is transformed, and some is lost.
Input x_t arrives at the current time step.
The model combines x_t with h_(t-1).
A new hidden state h_t is formed.
h_t becomes context for the next step.
Per-Step Output Versus Final State
A recurrent layer usually exposes two useful views of the sequence:
| Object | Meaning | Typical use |
|---|---|---|
output | Hidden representation at every time step | Token labeling, dense forecasting, inspection |
h_n | Final hidden state | Whole-sequence classification or summary |
PyTorch Example: Reading Hidden States
For a single-layer basic RNN, output[:, -1, :] and the final hidden state contain closely related information. But once you add multiple layers, bidirectionality, or LSTM cell states, it becomes important to know exactly which tensor you are reading.
What the Hidden State Learns
Short-range memory
- Recent values or tokens.
- Local momentum or trend.
- Immediate trigger conditions.
Abstract summaries
- Sentence sentiment clues.
- Machine operating mode.
- Forecasting context.
What it may lose
- Fine details from far in the past.
- Rare but important early events.
- Precise long-range structure.
Capacity and Bottlenecks
The hidden state is a bottleneck. A hidden size of 16 or 64 must encode whatever the task needs from possibly hundreds of time steps. If the task is simple, that is enough. If the sequence contains subtle long-distance relationships, this compression becomes a serious limitation.
Benefits
- Compact and reusable memory.
- Works with variable-length input.
- Lets the network summarize evolving context.
Tradeoffs
- Finite capacity.
- Can forget early information.
- Harder to optimize on long sequences.
Do not confuse the hidden state with a human-readable memory buffer. Individual dimensions usually do not map cleanly to interpretable concepts. The hidden state is a distributed representation, not a list of named facts.
“A bigger hidden size automatically fixes memory problems.” Increasing hidden size can help, but it does not solve training instability or guarantee that the optimization process will preserve useful early information.
Why This Leads to Gradient Problems
The hidden state must be updated again and again through the whole sequence. During training, gradients also have to flow backward through those repeated updates. That is why hidden-state reasoning naturally leads to the next lessons on vanishing gradients and exploding gradients.
The same core question will reappear in Volume 09 language modeling: how much earlier context should a model remember when interpreting the current token?
Knowledge Check
- Short Answer: What is the hidden state in an RNN? Answer: A learned internal memory vector passed from one time step to the next.
- True/False: The hidden state usually has the same value at every time step. Answer: False.
- Multiple Choice: Which tensor commonly stores representations for every step? (a)
output, (b) optimizer state, (c) loss history. Answer: (a). - Short Answer: Why is the hidden state called a bottleneck? Answer: Because a fixed-size vector must compress potentially large amounts of sequence context.
- True/False: Hidden states are mainly useful only for language tasks. Answer: False.
- Multiple Choice: For sequence classification, engineers often use: (a) a shuffled time step, (b) the final hidden summary, (c) image pooling only. Answer: (b).
- Short Answer: What two pieces of information are combined to create
h_t? Answer: The current input and the previous hidden state. - True/False: A larger hidden size always solves long-range dependency issues. Answer: False.
- Multiple Choice: Which next topic is most directly caused by repeatedly updating hidden states over long sequences? (a) Pooling, (b) Vanishing gradients, (c) Padding. Answer: (b).
- Short Answer: What does
output[:, -1, :]usually represent? Answer: The final time step representation for each sequence in the batch.
Key Takeaways
- The hidden state is the core memory mechanism that gives RNNs temporal context.
- It is a compressed representation, not a perfect record of the past.
- Per-step outputs and final states support different task types.
- Memory capacity and repeated updates create optimization challenges on long sequences.
- Next, Vanishing Gradient explains why early information often fades during training.
Hands-on idea: Ask students to inspect output[:, t, :] at different time steps and describe what the model should know at each position in a sentiment or forecasting task.
Discussion prompt: Debate whether a fixed-size state vector is a strength because it is compact or a weakness because it discards detail.
Recap: The hidden state is the evolving internal memory of a recurrent model, but its fixed size and repeated updates create important limits. Continue with Vanishing Gradient.