Weights — Module 6.1

How This Lesson Fits Module 6.1

Weights are the trainable numbers that let networks learn. After seeing the output layer, this lecture looks inside nn.Linear to show how predictions change when weights change.

Learning Objectives

By the end of this lesson, students should be able to:

Define weights as trainable parameters in neural layers.
Explain how weight matrices transform input vectors.
Connect weights to coefficients in linear and logistic regression.
Inspect PyTorch parameter shapes and gradients.
Describe initialization and regularization at a high level.
Recognize symptoms of poor weight scale.

Definition

A weight is a learned parameter that scales, selects, or mixes an input signal before it is passed to later computation.

Weights Are Learned Feature Mixers

In a dense layer, the weight matrix determines which input signals contribute to each output unit. Training changes those values to reduce loss. Small random initialization breaks symmetry so units can specialize; gradients tell each weight how a small change would affect the objective. Volume 05 coefficients were learned weights in simpler clothing.

Layer	Weight shape	Meaning
`nn.Linear(4, 3)`	`[3, 4]`	Three output units each read four inputs
Regression coefficient	`[features]`	One learned slope per feature
Embedding	`[vocab, dim]`	One vector per token/item
Convolution	`[out, in, kH, kW]`	Shared local pattern detectors
Attention projection	Matrix	Learns query/key/value spaces

PyTorch Practice

PyTorch exposes weights as parameters. Inspecting them early makes architecture and shape mistakes easier to catch.

import torch
from torch import nn

layer = nn.Linear(4, 3)
print(layer.weight.shape)  # [out_features, in_features]
print(layer.bias.shape)

x = torch.randn(2, 4)
y = layer(x)
loss = y.pow(2).mean()
loss.backward()

print(layer.weight.grad.shape)
print(layer.weight.data.mean().item())

How to Think About Weights

As coefficients

Positive increases score
Negative decreases score
Magnitude controls strength

As representation builders

Rows create new features
Matrices rotate and scale spaces
Depth compounds transformations

As regularized parameters

Weight decay discourages large values
Initialization sets starting scale
Gradients guide updates

Strengths and Tradeoffs

Useful because

Weights make networks adaptable to data.
Matrix operations train efficiently on GPUs.
Regularization can control weight growth.

Watch for

Too-large weights can saturate activations or destabilize training.
Too-small or symmetric weights can slow learning.
Many weights increase overfitting risk and memory cost.

How It Flows

1. Initialize

Start with random values scaled for the layer.

2. Forward

Use weights to transform inputs into activations.

3. Differentiate

Compute gradients with respect to loss.

4. Update

Optimizer changes weights using gradients and hyperparameters.

5. Regularize

Weight decay or other methods restrain unnecessary complexity.

Common Misconception

Do not initialize every weight to the same constant. Symmetric units receive the same gradients and fail to specialize, leaving capacity unused.

Knowledge Check

Short Answer: What is a weight? Answer: A learned parameter that scales or mixes input signals.
True/False: Dense-layer weights are usually stored in matrices. Answer: True.
Multiple Choice: nn.Linear(4,3) has weight shape: (a) [3,4], (b) [4,3], (c) [2,2]. Answer: (a).
Short Answer: What does .grad store? Answer: The gradient of loss with respect to a parameter.
True/False: Weight decay is related to L2 regularization. Answer: True.
Short Answer: Why random initialization? Answer: To break symmetry between units.
Multiple Choice: Large unchecked weights can: (a) destabilize training, (b) guarantee generalization, (c) remove loss. Answer: (a).
Short Answer: Volume 05 analogy for weights? Answer: Linear/logistic regression coefficients.
True/False: More weights always means better test performance. Answer: False.
Short Answer: What tool updates weights in training? Answer: An optimizer.

Key Takeaways

Weights are the core trainable parameters in neural networks.
Their shapes explain how layers transform tensors.
Initialization, gradients, and regularization determine how weights evolve.
Next, Bias Neuron explains the trainable offset paired with weights.

Trainer’s Guide

Hands-on idea: Have students print all named parameters in a small MLP and calculate the total parameter count by hand.

Discussion prompt: When could a model with fewer weights be preferable even if a larger model has lower training loss?

Recap: Weights are learned matrices that mix signals, build representations, and carry most of a network's capacity. Continue with Bias Neuron.