EfficientNet — Module 7.3

How This Lesson Fits the Module

ResNet let us go deeper safely, but researchers still scaled networks by intuition—more layers here, wider channels there. EfficientNet (Tan & Le, Google, 2019) asked: what is the optimal way to scale? Their answer, compound scaling, balances depth, width, and input resolution with a single coefficient—reaching higher accuracy with far fewer FLOPs and parameters.

ResNet (2015) — scale by depth (skip connections make it safe) ↓ EfficientNet (2019) — compound-scale depth + width + resolution together ↓ Related: MobileNet shares its efficient MBConv building block

Learning Objectives

By the end of this lesson, students should be able to:

Explain the three scaling dimensions: depth, width, and resolution.
Describe compound scaling and its single coefficient φ.
Identify the MBConv (inverted residual) block at EfficientNet’s core.
Compare the EfficientNet-B0 to B7 family and their accuracy/FLOP trade-offs.
Load EfficientNet via torchvision or timm.
Explain why EfficientNet is a strong default backbone today.

Definition — Compound Scaling

Compound scaling uniformly scales network depth (d = α^φ), width (w = β^φ), and resolution (r = γ^φ) with a single user coefficient φ, where α·β²·γ² ≈ 2. Instead of scaling one dimension arbitrarily, all three grow in balance, giving better accuracy per unit of compute.

Why Balance Beats One Dimension

Scaling only depth (like very deep ResNets) hits diminishing returns; only width captures fine features but misses high-level ones; only resolution needs matching capacity to use the extra pixels. EfficientNet showed that increasing all three proportionally is consistently more efficient. Starting from a NAS-designed baseline (EfficientNet-B0), scaling φ produces B1–B7.

Model	Input res	Params	ImageNet top-1	Relative cost
EfficientNet-B0	224	~5.3M	~77.1%	1×
EfficientNet-B3	300	~12M	~81.6%	~4×
EfficientNet-B4	380	~19M	~82.9%	~9×
EfficientNet-B7	600	~66M	~84.3%	~37×

For comparison: EfficientNet-B0 matches ResNet-50’s accuracy with roughly a fifth of the parameters.

Key Innovation

Compound scaling coefficient φ
MBConv (inverted residual) blocks
Squeeze-and-Excitation attention
NAS-designed B0 baseline

What It Improves

Accuracy per FLOP
Accuracy per parameter
Principled (not ad-hoc) scaling
A whole family from one recipe

Loading EfficientNet in PyTorch

import torch
import torch.nn as nn
from torchvision.models import efficientnet_b0, EfficientNet_B0_Weights

weights = EfficientNet_B0_Weights.IMAGENET1K_V1
model = efficientnet_b0(weights=weights)

# Swap the classifier head for a 10-class task
in_feats = model.classifier[1].in_features
model.classifier[1] = nn.Linear(in_feats, 10)
print(sum(p.numel() for p in model.parameters()))   # ~4.0M after head swap

# Alternatively, use timm for the full B0-B7 family + latest weights:
# pip install timm
import timm
m = timm.create_model("efficientnet_b3", pretrained=True, num_classes=10)
print(m.default_cfg["input_size"])                   # (3, 300, 300)

Module 7.1 Tie-In Higher B-numbers demand larger input resolution—a direct consequence of how convolution receptive fields and pooling interact with image size. Match the resolution to the model or accuracy drops.

Common Misconception: “EfficientNet is always the fastest model.”

Reality: EfficientNet optimizes accuracy per FLOP, not necessarily wall-clock latency. Its depthwise convolutions can be memory-bandwidth bound on some hardware; for raw mobile speed, MobileNet or hardware-specific models may win.

Critical Mistake — Wrong Input Resolution

Each EfficientNet variant was trained at a specific resolution (B0=224, B3=300, B7=600). Feeding B7 a 224×224 image throws away its main advantage. Always use the model’s intended resolution via weights.transforms() or timm’s default_cfg.

Knowledge Check

Short Answer: Name the three dimensions compound scaling balances. Answer: Depth, width, and resolution.
True/False: EfficientNet was introduced in 2019 by Google. Answer: True.
Multiple Choice: Compound scaling uses: (a) one coefficient φ, (b) three separate networks, (c) no scaling, (d) random search each time. Answer: (a).
Short Answer: What is EfficientNet’s core building block called? Answer: MBConv (inverted residual).
True/False: EfficientNet-B0 needs more parameters than ResNet-50 for similar accuracy. Answer: False—far fewer.
Multiple Choice: Which variant uses the largest input resolution? (a) B0, (b) B3, (c) B4, (d) B7. Answer: (d), 600.
Short Answer: Which attention mechanism sits inside MBConv blocks? Answer: Squeeze-and-Excitation.
True/False: EfficientNet always has the lowest latency of any model. Answer: False (it optimizes FLOPs, not always wall-clock).
Multiple Choice: The B0 baseline was found via: (a) hand tuning, (b) neural architecture search, (c) genetic art, (d) VGG rules. Answer: (b).
Short Answer: Give one reason to match a model’s intended input resolution. Answer: Accuracy drops / advantage lost if resolution mismatches training.

Key Takeaways

EfficientNet scales depth, width, and resolution together via one coefficient φ.
Compound scaling yields better accuracy per FLOP and per parameter.
The MBConv (inverted residual) + Squeeze-Excitation block is its core.
The B0–B7 family trades accuracy for compute along one recipe.
Next: MobileNet takes efficiency to phones and edge devices.

Trainer’s Guide

Exercise: Fine-tune EfficientNet-B0 and ResNet-50 on the same small dataset; compare accuracy, parameter count, and training time to make the efficiency argument concrete.

Discussion: Why might a “fewer FLOPs” model still be slower on a given GPU? Introduce the idea of memory bandwidth vs compute—a bridge to the MobileNet lecture.

Progress We now scale networks intelligently. Next we push efficiency to its extreme for mobile hardware. Continue to MobileNet.