SAM (Segment Anything) — Module 7.3

How This Lesson Fits the Module — Vol 07 Capstone

This is the final lecture of Volume 07. We began with LeNet reading digits and end with a model that can segment anything. SAM (Segment Anything Model, Meta AI, 2023) is a promptable, foundation-scale segmentation model: click a point, draw a box, or supply a mask, and it returns high-quality object masks—even for categories it never explicitly trained on. It fuses everything this volume taught: convolutional intuition, ViT backbones, and the promptable spirit of CLIP.

LeNet → ResNet — learn features with convolution ↓ ViT → CLIP — attention backbones + language-driven generalization ↓ SAM (2023) — a promptable segmentation foundation model

Learning Objectives

By the end of this lesson, students should be able to:

Explain what makes SAM a “promptable” foundation model.
Describe its three components: image encoder, prompt encoder, mask decoder.
Explain the role of the SA-1B dataset (1B masks) in its generality.
Perform point- and box-prompted segmentation with the segment_anything API.
Contrast SAM with task-specific models like Mask R-CNN.
Synthesize the full Vol 07 arc from LeNet to SAM.

Definition — Segment Anything Model (SAM)

SAM is a promptable segmentation model with three parts: a heavy ViT image encoder (run once per image), a lightweight prompt encoder (points, boxes, or masks), and a fast mask decoder that combines the two to output masks in milliseconds. Because the image is encoded once, many prompts can be answered interactively in real time.

Why SAM Generalizes

SAM was trained on SA-1B—over 1 billion masks across 11 million images—built with a data engine where SAM itself helped annotate. This scale gives it a strong notion of “objectness” that transfers zero-shot to new domains: medical scans, satellite imagery, microscopy. It segments based on prompts and visual structure, not a fixed class list.

Aspect	Mask R-CNN	SAM
Backbone	ResNet + FPN	ViT image encoder
Output	Masks for trained classes	Masks for any prompted object
Interaction	None (fixed)	Points, boxes, masks (promptable)
Generalization	Trained categories	Zero-shot to new domains
Year	2017	2023

Running SAM in PyTorch

# pip install segment-anything  (or use HuggingFace transformers)
import numpy as np
from segment_anything import sam_model_registry, SamPredictor

sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h.pth")
predictor = SamPredictor(sam)

predictor.set_image(image_rgb)               # encode image ONCE (heavy step)

# Prompt with a single foreground point (x, y)
point_coords = np.array([[420, 310]])
point_labels = np.array([1])                 # 1 = foreground, 0 = background
masks, scores, _ = predictor.predict(
    point_coords=point_coords,
    point_labels=point_labels,
    multimask_output=True,                   # returns 3 candidate masks
)
best = masks[scores.argmax()]                # [H, W] boolean mask

# Or prompt with a box [x1, y1, x2, y2] — decoder is fast, reuse encoding
box = np.array([100, 80, 540, 460])
masks_box, _, _ = predictor.predict(box=box, multimask_output=False)

Full-Module Tie-In SAM’s ViT encoder rests on convolution-era intuitions (patchify), its promptability echoes CLIP, and its masks refine the segmentation task—now interactive and universal.

Common Misconception: “SAM labels the objects it segments.”

Reality: SAM produces masks but does not assign semantic class names. Pair it with a classifier or CLIP to label the masked regions—SAM answers “where is the object,” not “what is it.”

Critical Mistake — Re-encoding the Image Per Prompt

The image encoder is the expensive part; the mask decoder is cheap. Call set_image() once, then issue many predict() prompts against that cached encoding. Re-running set_image() for every click throws away SAM’s real-time interactivity.

Knowledge Check

Short Answer: What are SAM’s three main components? Answer: Image encoder, prompt encoder, mask decoder.
True/False: SAM can segment objects from a single clicked point. Answer: True.
Multiple Choice: SAM’s image encoder is a: (a) LSTM, (b) ViT, (c) LeNet, (d) decision tree. Answer: (b).
Short Answer: What dataset was SAM trained on? Answer: SA-1B (~1 billion masks).
True/False: SAM assigns class labels to the masks it produces. Answer: False—it outputs masks, not labels.
Multiple Choice: Which step is the expensive one to run once per image? (a) prompt encoder, (b) mask decoder, (c) image encoder, (d) NMS. Answer: (c).
Short Answer: Name a prompt type SAM accepts. Answer: Point, box, or mask.
True/False: SAM generalizes zero-shot to new domains like medical imaging. Answer: True.
Multiple Choice: Compared to Mask R-CNN, SAM is: (a) class-fixed, (b) promptable and open-domain, (c) slower to prompt, (d) CNN-only. Answer: (b).
Short Answer: How would you get semantic labels for SAM’s masks? Answer: Pair it with a classifier or CLIP.

Key Takeaways

SAM is a promptable segmentation foundation model: point/box/mask → masks.
Encode the image once with a ViT; answer many prompts fast via the decoder.
Trained on SA-1B (~1B masks), it generalizes zero-shot across domains.
It outputs masks without labels—combine with CLIP/classifiers for names.
Volume 07 complete: from LeNet’s digits to universal, promptable vision.

Trainer’s Guide — Capstone Wrap-Up

Capstone project: Build an interactive tool—click a point, SAM returns a mask, then CLIP labels the masked region. Students combine three lectures into one working pipeline.

Synthesis discussion: Trace the volume’s arc: convolution (7.1) → tasks (7.2) → models (7.3). Ask students to place each model on a timeline and name the single advance it contributed.

Volume 07 Complete You have journeyed the entire history of vision models—LeNet, AlexNet, VGG, ResNet, EfficientNet, MobileNet, YOLO, Mask R-CNN, ViT, CLIP, and SAM. Next, we turn to data that unfolds over time: sequences. Continue to Vol. 08 — Sequence Models (RNN).