Image Segmentation — Module 7.2

How This Lesson Fits the Module

Module 7.2 has climbed a ladder of spatial precision. Classification gave one label per image; detection added coarse rectangles for "where." Both stop short of the object's true shape.

Image segmentation is the finest-grained vision task and the capstone of this module: it assigns a class label to every pixel, producing a mask that traces exact object boundaries. This lecture completes the task trilogy and hands off to Module 7.3, where architectures like Mask R-CNN and SAM implement these ideas at scale.

Learning Objectives

By the end of this lesson, students should be able to:

Define image segmentation and distinguish semantic, instance, and panoptic variants.
Contrast per-pixel masks with bounding boxes and whole-image labels.
Explain the encoder-decoder (U-Net) architecture and the role of skip connections.
Choose appropriate losses: pixel cross-entropy vs. Dice / IoU loss.
Compute and interpret pixel accuracy, mean IoU, and the Dice coefficient.
Load a pretrained segmentation model in torchvision and read its output.

What Is Image Segmentation?

Segmentation is classification performed at the pixel level. Where a classifier outputs one label and a detector outputs a handful of boxes, a segmentation model outputs a full-resolution map the same height and width as the input, where each pixel carries a class prediction. The result is a mask that follows the object's contour exactly—no wasted rectangle corners.

Definition — Image Segmentation

Image segmentation is the task of assigning a class label to every pixel of an image, producing a dense prediction map of shape (H, W). The output partitions the image into regions—a per-pixel classification—rather than a single label or a set of boxes.

Three Flavors of Segmentation

"Segmentation" covers three related but distinct problems. Confusing them leads to the wrong model and the wrong metric.

Semantic

Label every pixel by class
Does NOT separate instances
Two adjacent cars → one "car" blob
Models: U-Net, DeepLab, FCN

Instance

Separate mask per object
Car #1 vs. Car #2 distinguished
Ignores "stuff" (sky, road)
Model: Mask R-CNN

Panoptic

Semantic + instance combined
Every pixel gets class + instance id
"Things" and "stuff" together
Models: Panoptic FPN, Mask2Former

The Full Task Trilogy

Placing segmentation beside its siblings makes the progression of Module 7.2 concrete.

Task	Output granularity	Shape of prediction	Primary metric
Classification	Whole image	(C,) logits	Top-1 accuracy
Detection	Per object (box)	Set of (box, class, score)	mAP
Semantic Seg.	Per pixel	(C, H, W) logits	mean IoU
Instance Seg.	Per object (mask)	Set of (mask, class, score)	mask mAP

Boxes vs. Masks

The jump from detection to segmentation is the jump from rectangles to pixel-perfect outlines. Each buys precision at a cost.

Masks win when

Exact shape matters (medical scans, defect maps).
Objects are irregular or overlap heavily.
You need area/boundary measurements, not just location.

But masks cost more

Pixel-level labels are expensive and slow to annotate.
Dense outputs need more memory and compute.
Class imbalance (tiny foreground) complicates training.

Encoder-Decoder Architecture (U-Net)

Segmentation needs both what (semantic context, captured by downsampling) and where (precise location, lost during downsampling). The encoder-decoder design solves this: the encoder compresses the image into rich features; the decoder upsamples back to full resolution; and skip connections copy high-resolution detail from encoder to decoder so boundaries stay sharp.

1. Encoder

Conv + pooling shrink spatial size, grow channels (the backbone).

2. Bottleneck

Lowest resolution, richest semantic features.

3. Decoder

Transposed conv / upsampling restore resolution.

4. Skip Links

Encoder features fused into decoder to recover detail.

5. Head

1×1 conv → (C, H, W) per-pixel logits.

Definition — Skip Connection (in U-Net)

A skip connection concatenates (or adds) feature maps from an encoder stage directly into the matching decoder stage. This restores fine spatial detail that pooling discarded, giving crisp object edges. It is the same architectural idea as the residual skips from Module 6.1, applied across the resolution ladder.

import torch
import torch.nn as nn

def conv_block(in_ch, out_ch):
    return nn.Sequential(
        nn.Conv2d(in_ch, out_ch, 3, padding=1), nn.ReLU(inplace=True),
        nn.Conv2d(out_ch, out_ch, 3, padding=1), nn.ReLU(inplace=True),
    )

class MiniUNet(nn.Module):
    def __init__(self, n_classes):
        super().__init__()
        self.enc1 = conv_block(3, 64)
        self.enc2 = conv_block(64, 128)
        self.pool = nn.MaxPool2d(2)
        self.bottleneck = conv_block(128, 256)
        self.up   = nn.ConvTranspose2d(256, 128, 2, stride=2)
        self.dec2 = conv_block(256, 128)      # 256 = 128 (up) + 128 (skip)
        self.head = nn.Conv2d(128, n_classes, 1)   # per-pixel logits

    def forward(self, x):
        e1 = self.enc1(x)
        e2 = self.enc2(self.pool(e1))
        b  = self.bottleneck(self.pool(e2))
        d2 = self.up(b)
        d2 = torch.cat([d2, e2], dim=1)        # skip connection
        d2 = self.dec2(d2)
        return self.head(d2)                    # (B, C, H, W)

Loss Functions for Segmentation

Pixel-wise cross-entropy is the default, but it struggles when the foreground is tiny (a small tumor in a large scan). Overlap-based losses like Dice directly optimize the metric you care about.

Loss	How it works	Best for
Pixel Cross-Entropy	Classify each pixel independently	Balanced classes
Weighted CE	Up-weight rare classes	Mild imbalance
Dice Loss	1 − Dice coefficient (overlap)	Severe foreground/background imbalance
IoU (Jaccard) Loss	1 − soft IoU	Directly optimizing mIoU
Combo (CE + Dice)	Sum of both	Stable + imbalance-robust (common default)

import torch
import torch.nn.functional as F

def dice_loss(logits, target, eps=1e-6):
    # logits: (B, C, H, W); target: (B, H, W) class indices
    probs = F.softmax(logits, dim=1)
    target_1h = F.one_hot(target, logits.shape[1])       # (B,H,W,C)
    target_1h = target_1h.permute(0, 3, 1, 2).float()    # (B,C,H,W)
    dims = (0, 2, 3)
    inter = (probs * target_1h).sum(dims)
    union = probs.sum(dims) + target_1h.sum(dims)
    dice = (2 * inter + eps) / (union + eps)
    return 1 - dice.mean()

# Common default: combine pixel CE with Dice for stability + balance
def combo_loss(logits, target):
    return F.cross_entropy(logits, target) + dice_loss(logits, target)

Metrics: IoU and Dice

Segmentation is scored by how well predicted and ground-truth masks overlap. The two dominant metrics are close cousins.

Definition — IoU vs. Dice

mean IoU (Jaccard) = |A ∩ B| / |A ∪ B|, averaged over classes—the same overlap ratio used for boxes in detection, now applied to masks. Dice coefficient (F1) = 2|A ∩ B| / (|A| + |B|). Dice weights overlap more generously; the two always agree on ordering (Dice ≥ IoU) but differ in magnitude.

Metric	Formula	Range	Notes
Pixel Accuracy	correct px / total px	0–1	Misleading under imbalance
mean IoU	mean of \|A∩B\|/\|A∪B\|	0–1	Standard for semantic seg.
Dice / F1	2\|A∩B\|/(\|A\|+\|B\|)	0–1	Popular in medical imaging

Critical Mistake — Trusting Pixel Accuracy

If 98% of pixels are background, a model that predicts "background" everywhere scores 98% pixel accuracy while segmenting nothing. Always report mean IoU or Dice, which ignore the easy background dominance and measure real overlap.

Pretrained Segmentation in torchvision

import torch
from torchvision.models.segmentation import (
    deeplabv3_resnet50, DeepLabV3_ResNet50_Weights)

weights = DeepLabV3_ResNet50_Weights.DEFAULT
model = deeplabv3_resnet50(weights=weights)
model.eval()

preprocess = weights.transforms()
img = preprocess(pil_image).unsqueeze(0)     # (1, 3, H, W)

with torch.no_grad():
    out = model(img)["out"]                  # (1, 21, H, W) logits
    mask = out.argmax(dim=1)                 # (1, H, W) per-pixel class

print(mask.shape, mask.unique())             # classes present in image

Common Misconception: “Semantic segmentation tells apart individual objects.”

Reality: Semantic segmentation labels pixels by class only—three overlapping people become one "person" region. Separating individuals requires instance segmentation (e.g. Mask R-CNN).

Common Misconception: “Segmentation is just detection with tighter boxes.”

Reality: Boxes are always rectangles; masks follow arbitrary contours. Segmentation predicts a label for every pixel, which detection never does. Instance segmentation actually combines both: a box and a mask per object.

Architecture Callback The U-Net skip connections mirror the residual connections from Vol 06, and the encoder is a classification backbone—segmentation is built from parts you already know.

Knowledge Check

Short Answer: What does a segmentation model output for one image? Answer: A per-pixel class map of shape (C, H, W) → (H, W) after arg-max.
True/False: Semantic segmentation distinguishes individual object instances. Answer: False—that is instance segmentation.
Multiple Choice: Which combines semantic and instance labeling of every pixel? (a) panoptic, (b) binary, (c) detection, (d) classification. Answer: (a) panoptic.
Short Answer: What is the purpose of skip connections in U-Net? Answer: Recover high-resolution spatial detail lost during encoder downsampling, sharpening boundaries.
True/False: Pixel accuracy is a reliable metric when 98% of pixels are background. Answer: False—use mean IoU or Dice.
Multiple Choice: The Dice coefficient equals: (a) |A∩B|/|A∪B|, (b) 2|A∩B|/(|A|+|B|), (c) |A|+|B|, (d) |A∪B|. Answer: (b).
Short Answer: When is Dice loss preferred over plain cross-entropy? Answer: When there is severe foreground/background imbalance (tiny objects).
True/False: The encoder in a segmentation network is essentially a classification backbone. Answer: True.
Multiple Choice: Which Module 7.3 model performs instance segmentation? (a) LeNet, (b) Mask R-CNN, (c) VGG16, (d) AlexNet. Answer: (b).
Short Answer: Which promptable foundation model in Module 7.3 segments almost anything zero-shot? Answer: SAM (Segment Anything).

Key Takeaways

Segmentation is per-pixel classification—the finest-grained vision task.
Semantic labels classes, instance separates objects, panoptic does both.
Encoder-decoder (U-Net) with skip connections balances semantics and precise localization.
Use Dice / IoU loss (or CE+Dice) when foreground is small; avoid pixel-accuracy metrics.
Report mean IoU and Dice—the mask analogues of detection's IoU.
Module complete: this trilogy feeds the Module 7.3 architectures.

Trainer’s Guide

Visual demo: Overlay a semantic mask and an instance mask on the same crowd photo—students instantly see "one blob" vs. "separated people."

Exercise: Have students compute IoU and Dice for the same predicted mask by hand to feel why Dice ≥ IoU.

Bridge to 7.3: Run pretrained DeepLabV3 (semantic) and Mask R-CNN (instance) side by side to motivate the next module.

Module 7.2 Complete You have mastered the three vision tasks—classification, detection, and segmentation. Next, Module 7.3 studies the landmark architectures that implement them, beginning with LeNet.