Forward Propagation — Module 6.1

How This Lesson Fits the Module

You have built an ANN from input, hidden, and output layers. Forward propagation is the inference path: data flows layer by layer to produce predictions and loss.

Every training step starts here. Backpropagation reverses this flow; optimizers act on the gradients it produces.

Learning Objectives

By the end of this lesson, students should be able to:

Trace tensor shapes through linear layers and activations.
Compute a single neuron output as weighted sum + bias + activation.
Implement forward pass in PyTorch with nn.Module.
Distinguish training forward pass (with loss) from inference (model.eval()).
Explain why activations must be stored for backpropagation.

Layer-by-Layer Data Flow

For a dense layer: z = Wx + b, then a = σ(z) where σ is an activation like ReLU or softmax on the output head.

Definition — Forward Propagation

Forward propagation (forward pass) evaluates the network function from inputs to outputs by applying each layer’s transformation in sequence. No gradients are required for the math—PyTorch builds the computation graph during training so gradients can flow backward later.

Stage	Operation	Shape Example (batch=32, in=784, hidden=128)
Input	Raw features	(32, 784)
Linear 1	`x @ W.T + b`	(32, 128)
ReLU	`max(0, z)`	(32, 128)
Linear 2	Logits	(32, 10)
Loss	Cross-entropy vs labels	scalar

PyTorch Forward Pass

import torch
import torch.nn as nn
import torch.nn.functional as F

class MLP(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(784, 128)
        self.fc2 = nn.Linear(128, 10)

    def forward(self, x):
        x = x.view(x.size(0), -1)      # flatten: (B, 784)
        x = F.relu(self.fc1(x))        # (B, 128)
        logits = self.fc2(x)           # (B, 10) — no softmax inside
        return logits

model = MLP()
x = torch.randn(32, 1, 28, 28)
y = torch.randint(0, 10, (32,))
logits = model(x)
loss = F.cross_entropy(logits, y)  # softmax + NLL fused for stability

Design Note Apply softmax inside the loss for classification; export logits at inference unless you need probabilities for calibration.

Training vs Inference Mode

model.train() enables dropout and batch norm training behavior. model.eval() disables dropout and uses running statistics for batch norm. Forward math is the same for linear layers; stochastic layers differ.

Training Forward

model.train()
Loss computed every batch
Graph built for backward()
Dropout active (see Dropout)

Inference Forward

model.eval() + torch.no_grad()
No loss, no backward
Deterministic (no dropout)
Faster, lower memory

Common Misconception: “Forward pass and prediction are the same thing.”

Reality: Forward pass produces logits or embeddings; prediction adds argmax, thresholding, or decoding. Loss is only computed during training forward passes.

Common Misconception: “Softmax must be the last layer in the model.”

Reality: CrossEntropyLoss expects raw logits. Putting softmax before the loss hurts numerical stability and is redundant.

Knowledge Check

Short Answer: Formula for one dense layer before activation? Answer: z = Wx + b.
True/False: Forward propagation computes gradients. Answer: False—backward pass does.
Multiple Choice: After Linear(784,128) with batch 32, output shape: (a) (128,32), (b) (32,128), (c) (32,784), (d) scalar. Answer: (b).
Short Answer: Why store activations during forward pass? Answer: Backprop needs them to compute gradients w.r.t. weights and earlier layers.
True/False: model.eval() changes linear layer weights. Answer: False—only behavior of certain modules like Dropout/BatchNorm.
Multiple Choice: CrossEntropyLoss expects: (a) probabilities, (b) logits, (c) one-hot only, (d) labels as floats. Answer: (b).
Short Answer: What does view(x.size(0), -1) do for MNIST? Answer: Flattens each image to a 784-vector while keeping batch dimension.
True/False: torch.no_grad() is recommended for inference. Answer: True.
Multiple Choice: Pass that reverses forward flow: (a) Backpropagation, (b) Epoch, (c) Batch, (d) Sigmoid. Answer: (a).
Short Answer: Name one activation used in the example MLP. Answer: ReLU.

Key Takeaways

Forward propagation maps inputs to logits through layers and activations.
Track tensor shapes—most bugs are shape mismatches.
Use logits + CrossEntropyLoss for classification training.
train() vs eval() matters for dropout and batch norm.
Next: Backpropagation — how loss signals flow backward.

Trainer’s Guide

Hands-on idea: Print intermediate shapes in forward() for a broken model; students fix a transpose error.

Whiteboard: Draw one neuron with three inputs, weights, bias, and ReLU for a numeric walkthrough.

What’s Next Forward pass creates the graph for gradient flow — continue to Backpropagation.