The previous lecture, image classification, answered "what is in this image?" but assumed a single dominant subject. Real scenes contain many objects at many scales—a street photo has cars, pedestrians, signs, and traffic lights all at once.
Object detection is the second vision task in Module 7.2. It answers "what AND where?" by predicting a variable number of bounding boxes, each with a class label and a confidence score. It sits between classification (one global label) and segmentation (per-pixel labels), and it introduces the machinery—anchors, IoU, NMS, and mAP—that powers the Module 7.3 detectors like YOLO and Mask R-CNN.
Learning Objectives
By the end of this lesson, students should be able to:
- Define object detection and contrast its output with classification and segmentation.
- Describe bounding-box representations and compute Intersection over Union (IoU).
- Explain the roles of the classification head and the box-regression head.
- Distinguish one-stage (YOLO/SSD) from two-stage (Faster R-CNN) detectors.
- Apply Non-Maximum Suppression (NMS) to remove duplicate detections.
- Interpret mean Average Precision (mAP) as the standard detection metric.
What Is Object Detection?
Object detection localizes and classifies every object of interest in an image simultaneously. Instead of a single label, the model emits a set of predictions—possibly zero, possibly hundreds—where each prediction is a tuple of (box coordinates, class, confidence). The variable-length output is what makes detection fundamentally harder than classification.
Object detection is the task of predicting a set of bounding boxes that tightly enclose objects, together with a class label and confidence score for each box. Formally, the model outputs {(bi, ci, si)} where b is a box, c a class, and s a confidence in [0,1].
Bounding Boxes
A bounding box is an axis-aligned rectangle described by four numbers. Two conventions dominate, and mixing them is a classic source of bugs.
| Format | Values | Used By |
|---|---|---|
xyxy | (x_min, y_min, x_max, y_max) | torchvision, Pascal VOC |
xywh | (x_min, y_min, width, height) | COCO annotations |
cxcywh | (center_x, center_y, width, height) | YOLO, DETR |
Feeding xywh boxes to a function expecting xyxy produces silently wrong IoU values and garbage NMS results—no error is raised. Always normalize to one format early; use torchvision.ops.box_convert to convert explicitly.
Intersection over Union (IoU)
IoU measures how well a predicted box overlaps a ground-truth box. It is the currency of detection: it decides whether a prediction counts as a match, drives NMS, and defines the thresholds in mAP.
Intersection over Union is the area of overlap between two boxes divided by the area of their union: IoU = |A ∩ B| / |A ∪ B|. It ranges from 0 (no overlap) to 1 (perfect overlap). A prediction is typically a "true positive" if IoU with a ground-truth box exceeds a threshold (e.g. 0.5).
Detection Architectures: One-Stage vs. Two-Stage
Detectors split into two families that trade speed against accuracy. Both attach a classification head (what class) and a box-regression head (refine coordinates) onto a shared classification backbone.
Two-Stage (Faster R-CNN)
- Stage 1: propose regions (RPN)
- Stage 2: classify + refine each
- Higher accuracy
- Slower—good for offline / precision
One-Stage (YOLO / SSD)
- Dense prediction in a single pass
- No separate proposal step
- Real-time speed
- Great for video / edge devices
Transformer (DETR)
- Set prediction, no anchors/NMS
- Bipartite matching loss
- End-to-end, elegant
- Data-hungry, slower to converge
Non-Maximum Suppression (NMS)
Detectors emit many overlapping boxes around the same object. NMS keeps the highest-confidence box and discards its near-duplicates (those with IoU above a threshold), leaving one clean box per object.
Using a Pretrained Detector in torchvision
torchvision ships production-ready detectors with pretrained COCO weights. Here we load Faster R-CNN and run inference.
Evaluating Detectors: mean Average Precision
Detection cannot use plain accuracy—there is no fixed number of predictions to be "right" about. The standard metric is mAP, which summarizes the precision-recall curve across confidence thresholds and IoU thresholds.
Average Precision (AP) is the area under the precision-recall curve for one class at a given IoU threshold. mAP averages AP over all classes. COCO reports mAP@[.5:.95]—the mean of AP computed at ten IoU thresholds from 0.50 to 0.95—rewarding both correct classification and tight localization.
| Metric | Meaning | Notes |
|---|---|---|
| [email protected] | AP averaged over classes at IoU ≥ 0.5 | Pascal VOC style, lenient localization |
| mAP@[.5:.95] | Mean of AP over IoU 0.5→0.95 | COCO primary metric, strict |
| APsmall/med/large | AP by object size | Exposes small-object weakness |
| Recall@k | Fraction of objects found in top-k | Useful for proposal quality |
Boxes vs. Masks
Detection stops at rectangles. The next lecture, segmentation, replaces boxes with pixel-perfect masks. Understanding the trade-off clarifies when each task is appropriate.
Bounding boxes are great when
- You only need object location and count.
- Speed matters (real-time video).
- Labels are cheap—annotators just drag rectangles.
But boxes fall short when
- Object shape is irregular (a box wastes area).
- Objects overlap heavily.
- You need exact pixel boundaries—use masks.
Reality: The number of detections is variable and data-dependent. Internally models score many candidates, but after confidence thresholding and NMS the final count varies per image—from zero to hundreds.
Reality: Confidence reflects classification certainty, not localization quality. A box can be 0.99 confident yet loosely placed (low IoU). mAP@[.5:.95] penalizes exactly this by rewarding tight boxes.
Knowledge Check
- Short Answer: What does each detection prediction contain? Answer: A bounding box, a class label, and a confidence score.
- True/False: Object detection produces a variable number of outputs per image. Answer: True.
- Multiple Choice: IoU is defined as: (a) intersection minus union, (b) intersection / union, (c) union / intersection, (d) area of the box. Answer: (b).
- Short Answer: What does NMS remove? Answer: Duplicate/overlapping boxes for the same object, keeping the highest-confidence one.
- True/False: One-stage detectors like YOLO are generally faster than two-stage detectors like Faster R-CNN. Answer: True.
- Multiple Choice: The COCO primary metric is: (a) top-5 accuracy, (b) mAP@[.5:.95], (c) Dice, (d) F1. Answer: (b).
- Short Answer: Which two heads sit on a detection backbone? Answer: A classification head and a box-regression head.
- True/False: The
xyxyandxywhbox formats are interchangeable without conversion. Answer: False—they must be converted. - Multiple Choice: Which Module 7.3 model is a one-stage real-time detector? (a) YOLO, (b) LeNet, (c) CLIP, (d) VGG16. Answer: (a).
- Short Answer: Why can't plain accuracy evaluate a detector? Answer: There is no fixed set of predictions; you must match boxes to ground truth via IoU, hence mAP.
Key Takeaways
- Detection = classification + localization: a set of (box, label, score) predictions.
- IoU quantifies box overlap and underpins matching, NMS, and mAP.
- NMS collapses overlapping boxes into one detection per object.
- One-stage detectors trade a little accuracy for real-time speed; two-stage maximize accuracy.
- mAP@[.5:.95] is the standard metric—it rewards tight, correctly classified boxes.
- Next: Image Segmentation upgrades boxes to per-pixel masks.
Whiteboard: Draw two overlapping rectangles and compute IoU by hand—students internalize the ratio far better than from a formula alone.
Live demo: Run pretrained Faster R-CNN and YOLO on the same webcam feed; contrast latency vs. box quality.
Exercise: Give students raw boxes with duplicates and have them implement NMS before revealing torchvision.ops.nms.