Bias (Neuron) — Module 6.1

How This Lesson Fits Module 6.1

Bias terms complete the weighted-sum formula introduced in Weights. They look small, but they let neurons shift thresholds and model nonzero baselines before activations such as sigmoid or ReLU are applied.

Learning Objectives

By the end of this lesson, students should be able to:

Define bias as a trainable additive offset.
Explain why bias terms shift activation thresholds.
Relate neural network bias to regression intercepts.
Inspect bias parameters in PyTorch layers.
Know when layers may omit bias because another component supplies an offset.
Avoid misconceptions about bias and dataset bias.

Definition

A bias term is a trainable value added to a weighted sum so a unit can activate even when input evidence is zero or centered.

Bias Moves the Boundary

Weights control direction and sensitivity; bias controls offset. In a single neuron, changing the bias shifts where the activation turns on. In a classifier, that means the decision boundary can move without changing its orientation. This is the same role played by an intercept in linear regression or logistic regression.

Context	Bias role	Example
Linear regression	Intercept	Prediction when features are zero
Perceptron	Threshold shift	Boundary moves left/right
Dense layer	One offset per output unit	`nn.Linear(..., bias=True)`
After BatchNorm	Sometimes redundant	Bias may be disabled before normalization
Output head	Base-rate adjustment	Initial class imbalance signal

PyTorch Practice

By default, nn.Linear includes a bias vector with one value for each output unit.

import torch
from torch import nn

layer = nn.Linear(3, 2, bias=True)
print(layer.weight.shape)  # [2, 3]
print(layer.bias.shape)    # [2]

with torch.no_grad():
    layer.weight.zero_()
    layer.bias[:] = torch.tensor([1.5, -0.5])

x = torch.randn(4, 3)
print(layer(x))  # output comes entirely from bias

Bias vs Weight

Weights

Scale input features
Determine direction
Depend on input values

Bias

Adds offset
Moves threshold or baseline
Exists even when input is zero

Together

Create affine transformation
Feed activation functions
Are learned by gradients

Strengths and Tradeoffs

Useful because

Allows flexible decision boundaries and nonzero baselines.
Improves expressiveness with very little parameter cost.
Makes units less dependent on perfectly centered inputs.

Watch for

Can be confused with social/statistical bias in data, which is different.
May be unnecessary before normalization layers that include affine offsets.
Incorrect initialization in output heads can slow early learning on imbalanced data.

How It Flows

1. Mix

Weights compute a weighted sum of inputs.

2. Shift

Bias adds a trainable offset.

3. Activate

The shifted value passes through sigmoid, tanh, ReLU, or another activation.

4. Learn

Gradients update both weights and bias values.

Common Misconception

A bias neuron is not the same as biased data or unfair model behavior. The term here means a mathematical intercept; data bias is an ethical and statistical issue handled by dataset design, evaluation, and governance.

Knowledge Check

Short Answer: What does a bias term add? Answer: A trainable offset.
True/False: Bias terms are learned parameters. Answer: True.
Multiple Choice: Bias is most like: (a) intercept, (b) learning rate, (c) batch size. Answer: (a).
Short Answer: In nn.Linear(3,2), bias shape is what? Answer: [2].
True/False: Bias changes the orientation of a linear boundary by itself. Answer: False; it shifts the boundary.
Short Answer: Why can bias help if inputs are zero-centered? Answer: It lets units learn nonzero activation thresholds.
Multiple Choice: PyTorch linear layers include bias by default: (a) yes, (b) no, (c) only for images. Answer: (a).
Short Answer: When might bias be disabled? Answer: Before a normalization layer with its own affine offset.
True/False: Mathematical bias and dataset bias mean the same thing. Answer: False.
Short Answer: What formula combines weights and bias? Answer: w·x + b.

Key Takeaways

Bias is the trainable offset in an affine transformation.
It shifts thresholds and baselines without changing weight direction.
Bias terms are small but important for expressive layers.
Next, Sigmoid introduces the first activation function in the sequence.

Trainer’s Guide

Hands-on idea: Use a one-feature logistic example and let students move only the bias to see how many points cross the decision threshold.

Discussion prompt: Why do small mathematical details like an intercept matter so much in model behavior?

Recap: Bias terms shift neuron responses, giving weighted sums the flexible baseline they need before activation. Continue with Sigmoid.