← Master Index
Vol. 06 Module 6.1 Lecture

Sigmoid

Neural Network Foundations

How This Lesson Fits Module 6.1

Sigmoid is the classic activation that turns any real-valued score into a value between 0 and 1. It connects biased weighted sums to probability-like outputs and prepares students to compare tanh, ReLU, and softmax.

Learning Objectives

By the end of this lesson, students should be able to:

  • Define the sigmoid function and its output range.
  • Explain why sigmoid is useful for binary probabilities.
  • Describe saturation and vanishing-gradient risk.
  • Use BCEWithLogitsLoss correctly in PyTorch.
  • Distinguish training logits from inference probabilities.
  • Choose sigmoid for binary or multilabel outputs, not most hidden layers.
Definition

The sigmoid function maps a scalar z to 1 / (1 + e-z), producing a value strictly between 0 and 1.

From Logit to Probability-Like Score

Sigmoid compresses large negative scores toward 0 and large positive scores toward 1. That makes it natural for binary classification and independent multilabel outputs. In hidden layers, however, saturation can slow learning because extreme inputs produce tiny gradients. Modern networks usually prefer ReLU-family activations inside the model and reserve sigmoid for output interpretation when appropriate.

Input logitSigmoid outputInterpretation
Large negativeNear 0Strong evidence for negative class
00.5Boundary/uncertain point
Large positiveNear 1Strong evidence for positive class
Multilabel vectorIndependent 0–1 valuesEach label can be on/off
Hidden unitSquashed activationCan saturate and slow gradients

PyTorch Practice

For training binary classifiers, pass raw logits to BCEWithLogitsLoss; apply sigmoid later for reporting probabilities.

import torch from torch import nn logits = torch.tensor([-3.0, 0.0, 2.0]) probs = torch.sigmoid(logits) print(probs) target = torch.tensor([0.0, 1.0, 1.0]) loss_fn = nn.BCEWithLogitsLoss() loss = loss_fn(logits, target) print(loss.item()) pred = (probs >= 0.5).long() print(pred)

Where Sigmoid Fits

Good fit

  • Binary output probability
  • Independent multilabel outputs
  • Gating mechanisms in some architectures

Poor default

  • Deep hidden layers
  • Multiclass exclusive classes
  • Large unnormalized hidden values

PyTorch habit

  • Train with logits
  • Use stable combined losses
  • Convert to probabilities for metrics or UI

Strengths and Tradeoffs

Useful because

  • Interpretable 0–1 output range.
  • Natural for binary and multilabel decisions.
  • Smooth and differentiable.

Watch for

  • Saturates for large positive or negative inputs.
  • Outputs are not zero-centered.
  • Can cause vanishing gradients in deep hidden stacks.

How It Flows

1. Score

Output layer emits a raw logit.

2. Train

Stable loss consumes the logit directly.

3. Convert

Sigmoid maps logit to a 0–1 value.

4. Threshold

A business or metric threshold turns probability into a label.

Common Misconception

Do not put Sigmoid before BCEWithLogitsLoss. That loss already combines sigmoid and binary cross-entropy in a numerically stable way.

Knowledge Check

  1. Short Answer: What range does sigmoid output? Answer: Between 0 and 1.
  2. True/False: Sigmoid is smooth and differentiable. Answer: True.
  3. Multiple Choice: Sigmoid is most common for: (a) binary probability, (b) exclusive multiclass logits, (c) file paths. Answer: (a).
  4. Short Answer: What is sigmoid(0)? Answer: 0.5.
  5. True/False: Saturation can make gradients tiny. Answer: True.
  6. Short Answer: Which PyTorch loss pairs with raw binary logits? Answer: BCEWithLogitsLoss.
  7. Multiple Choice: For hidden layers today, sigmoid is often replaced by: (a) ReLU, (b) CSV, (c) argmax. Answer: (a).
  8. Short Answer: Why use sigmoid for multilabel? Answer: Labels are independent rather than mutually exclusive.
  9. True/False: Sigmoid outputs are zero-centered. Answer: False.
  10. Short Answer: What is a common default threshold? Answer: 0.5, though it may be tuned.

Key Takeaways

  • Sigmoid maps logits to 0–1 probability-like values.
  • Use it mainly for binary or multilabel output interpretation.
  • Train binary classifiers with raw logits and stable PyTorch losses.
  • Next, Tanh compares another saturating activation with a zero-centered range.
Trainer’s Guide

Hands-on idea: Plot sigmoid values from -8 to 8 and ask students to mark where gradients are largest and smallest.

Discussion prompt: Why can a function that is excellent for probabilities be a poor default for deep hidden layers?

Recap: Sigmoid is useful for binary probability interpretation, but saturation limits its role inside deep networks. Continue with Tanh.