Artificial Neural Network (ANN) — Module 6.1

How This Lesson Fits Module 6.1

This opening lecture is the bridge from ElasticNet and regularization to deep learning. In Volume 05 you learned to control linear and tree-based models; in Volume 06 you keep the same discipline, but replace hand-shaped model families with layered differentiable functions trained by gradients.

Learning Objectives

By the end of this lesson, students should be able to:

Define an artificial neural network as a composition of parameterized layers.
Relate neurons, weights, bias terms, activations, and loss to classical ML concepts.
Explain why depth creates reusable intermediate representations.
Build a minimal feedforward network in PyTorch.
Identify where regularization, validation, and optimization reappear in deep learning.
Prepare for the layer-by-layer lectures that follow in Module 6.1.

Definition

An artificial neural network is a differentiable model that maps inputs to outputs through layers of weighted sums, bias terms, nonlinear activation functions, and trainable parameters optimized against a loss function.

From Penalized Linear Models to Learned Representations

A linear model learns one weighted combination of features. An ANN learns many weighted combinations, transforms them with nonlinear functions, and stacks the result. The early lectures in this module unpack that stack: the input layer receives tensors, hidden layers learn representations, and the output layer matches the prediction task. The same safeguards from Volume 05 still matter: clean splits, scaled inputs, validation loss, and regularization.

ANN part	Role	Volume 05 analogy
Input tensor	Carries features into the model	Feature matrix `X`
Weights	Scale and mix signals	Regression coefficients
Bias	Shifts activation thresholds	Intercept term
Activation	Adds nonlinearity	Feature transformation
Loss	Defines training objective	MSE, log loss, regularized objective

PyTorch Practice

This small network shows the whole template: layers in __init__, tensor flow in forward, and a task-specific output dimension.

import torch
from torch import nn

class TinyANN(nn.Module):
    def __init__(self, in_features: int, num_classes: int):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(in_features, 32),
            nn.ReLU(),
            nn.Linear(32, 16),
            nn.ReLU(),
            nn.Linear(16, num_classes),
        )

    def forward(self, x):
        return self.net(x)

model = TinyANN(in_features=20, num_classes=3)
x = torch.randn(8, 20)
logits = model(x)
print(logits.shape)  # torch.Size([8, 3])

ANN Anatomy

Classical ML carryovers

Training/validation/test discipline
Loss functions and optimization
Regularization as capacity control

Deep learning additions

Layered representation learning
Nonlinear activations
Automatic differentiation through many parameters

Engineering habit

Track tensor shapes
Match output layer to task
Inspect train and validation curves together

Why Neural Networks Matter

Useful because

Learn features instead of requiring every interaction to be manually engineered.
Scale from tabular examples to images, audio, text, and sequences.
Reuse the same training loop across many architectures.

Watch for

More data- and compute-hungry than many classical baselines.
Sensitive to shapes, initialization, learning rate, and loss selection.
Can overfit dramatically without validation and regularization.

How It Flows

1. Choose representation

Convert examples into tensors with stable shapes and numerical scale.

2. Forward pass

Layers transform inputs into logits, scores, or predictions.

3. Compute loss

The objective compares model output with the target.

4. Backpropagate

Autograd computes gradients for every trainable parameter.

5. Update

An optimizer moves weights to reduce future loss.

Common Misconception

A neural network is not automatically better than a classical model. If the dataset is small, mostly tabular, or poorly validated, a simple regularized baseline can outperform a deep model and be easier to maintain.

Knowledge Check

Short Answer: What is the basic job of an ANN? Answer: Learn a mapping from input tensors to outputs through trainable layers.
True/False: Weights in a neural network are learned from data. Answer: True.
Multiple Choice: Which component adds nonlinearity? (a) activation, (b) batch size, (c) dataset split. Answer: (a).
Short Answer: Volume 05 concept most closely related to weight decay? Answer: L2/Ridge regularization.
True/False: The output layer should be designed independently of the task. Answer: False.
Short Answer: Why stack hidden layers? Answer: To learn progressively richer intermediate representations.
Multiple Choice: PyTorch models usually subclass: (a) nn.Module, (b) DataFrame, (c) GridSearchCV. Answer: (a).
Short Answer: What does a loss function provide? Answer: A scalar training objective.
True/False: Validation loss still matters in deep learning. Answer: True.
Short Answer: Name one risk of deep networks. Answer: Overfitting, shape bugs, high compute cost, or unstable training.

Key Takeaways

ANNs extend the regularized modeling mindset from Volume 05 into layered differentiable functions.
Weights, bias terms, activations, and losses are the core vocabulary of Module 6.1.
Depth is valuable because it learns intermediate representations, not because it is magic.
Next, Perceptron reduces the network idea to one trainable decision unit.

Trainer’s Guide

Hands-on idea: Have students implement TinyANN, print every parameter shape, then compare those shapes to a linear model coefficient vector.

Discussion prompt: What parts of Volume 05 become more important, not less important, once a model has thousands or millions of parameters?

Recap: An ANN is a layered, trainable function that keeps the same validation and regularization responsibilities you practiced in classical ML. Continue with the Perceptron.