Hidden Layer — Module 6.1

How This Lesson Fits Module 6.1

Hidden layers are where neural networks earn their name: they transform inputs into learned representations. This lecture connects the input layer to output decisions through stacked nonlinear computation.

Learning Objectives

By the end of this lesson, students should be able to:

Define a hidden layer and distinguish it from input and output layers.
Explain how hidden layers learn intermediate features.
Describe why nonlinear activations are required for depth to help.
Choose reasonable hidden widths for starter networks.
Implement a multilayer perceptron in PyTorch.
Recognize overfitting risks from excessive capacity.

Definition

A hidden layer is an internal network layer whose activations are not direct inputs or final predictions; they are learned intermediate representations used by later layers.

Representation Learning

In classical ML, feature engineering often creates interactions by hand. Hidden layers learn those interactions from data. The first hidden layer may combine raw features into simple patterns; later layers combine those patterns into more abstract signals. Without nonlinear activations such as ReLU or tanh, stacked linear layers collapse into one linear layer.

Design choice	Effect	Practical note
Width	Number of units in a layer	Start modest; increase with data and complexity
Depth	Number of hidden layers	More depth learns hierarchy but is harder to optimize
Activation	Nonlinear transform	Required for nonlinear decision boundaries
Dropout/weight decay	Regularization	Controls overfitting in high-capacity layers
Normalization	Stabilizes distributions	Often used in deeper networks

PyTorch Practice

This multilayer perceptron uses two hidden layers. Each linear transformation is followed by a nonlinearity before the final prediction layer.

import torch
from torch import nn

class MLP(nn.Module):
    def __init__(self, in_features, num_classes):
        super().__init__()
        self.layers = nn.Sequential(
            nn.Linear(in_features, 64),
            nn.ReLU(),
            nn.Linear(64, 32),
            nn.ReLU(),
            nn.Linear(32, num_classes),
        )

    def forward(self, x):
        return self.layers(x)

model = MLP(20, 4)
print(model(torch.randn(5, 20)).shape)

Hidden Layer Intuition

Early hidden layers

Detect simple feature combinations
Sensitive to raw input quality
Often wider than later layers

Later hidden layers

Combine earlier patterns
Represent task-specific abstractions
Feed final output decisions

No hidden layers

Equivalent to linear/logistic regression for many tasks
Useful baseline
Limited nonlinear capacity

Strengths and Tradeoffs

Useful because

Learn feature interactions automatically.
Can model complex nonlinear relationships.
Reusable design pattern across tabular, vision, and language tasks.

Watch for

Too many units can memorize small datasets.
Deep stacks can suffer unstable gradients without good design.
Interpretability is harder than in simple linear models.

How It Flows

1. Receive

Take activations from the previous layer.

2. Mix

Apply a trainable linear transformation.

3. Activate

Use a nonlinear function to keep depth expressive.

4. Pass

Send the new representation to the next layer.

Common Misconception

Adding layers without nonlinear activations does not create a deep nonlinear model. Several linear layers in a row are mathematically equivalent to one linear layer.

Knowledge Check

Short Answer: What is a hidden layer? Answer: An internal layer that learns intermediate representations.
True/False: Hidden activations are usually final predictions. Answer: False.
Multiple Choice: Width means: (a) units per layer, (b) learning rate, (c) target count. Answer: (a).
Short Answer: Why use nonlinear activations? Answer: To let stacked layers model nonlinear functions.
True/False: More capacity can increase overfitting risk. Answer: True.
Short Answer: Name one regularizer for hidden layers. Answer: Dropout or weight decay.
Multiple Choice: A common hidden activation is: (a) ReLU, (b) CSV, (c) MSE target. Answer: (a).
Short Answer: What do later hidden layers combine? Answer: Patterns learned by earlier layers.
True/False: Hidden layers replace all need for validation. Answer: False.
Short Answer: What does an MLP stand for? Answer: Multilayer perceptron.

Key Takeaways

Hidden layers learn intermediate representations.
Nonlinearity is what makes depth expressive.
Depth and width are capacity choices that require validation.
Next, the Output Layer adapts representations to a prediction task.

Trainer’s Guide

Hands-on idea: Ask students to remove ReLU from the MLP, then reason about why the network's expressive power changes before running it.

Discussion prompt: When would a shallow model with excellent feature engineering beat a deeper network?

Recap: Hidden layers transform inputs into learned features, but their capacity must be controlled and validated. Continue with Output Layer.