CPU vs GPU — Module 6.3

How This Lesson Fits the Module

Module 6.2 taught you how to train—training loops, checkpoints, and distributed training across machines. Module 6.3 answers where that work runs: on the CPU or on the GPU. Every matrix multiply in backpropagation is a parallel workload; understanding the hardware split is what turns a notebook that “works on CPU” into one that trains in minutes instead of days.

This opening lecture frames the rest of the module—CUDA cores, VRAM, tensor cores, and numeric precision—as layers of the same stack.

Learning Objectives

By the end of this lesson, students should be able to:

Contrast CPU and GPU architectures in terms of core count, clock speed, and memory bandwidth.
Explain why deep learning favors GPUs for training and when CPUs remain the right choice.
Move tensors and models to the correct device in PyTorch with .to(device).
Identify host–device transfer bottlenecks and minimize unnecessary CPU↔GPU copies.
Read basic GPU utilization signals (nvidia-smi, PyTorch profiler) to confirm the GPU is actually working.

Two Processors, Two Design Goals

A CPU (Central Processing Unit) is optimized for low-latency sequential work: a few powerful cores (often 8–64) with large caches, branch prediction, and complex control logic. A GPU (Graphics Processing Unit) is optimized for high-throughput parallel work: thousands of simpler cores that execute the same instruction on many data elements at once—exactly the pattern of a batched matrix multiply or convolution.

Dimension	CPU	GPU (NVIDIA datacenter example)
Core philosophy	Few fast, general-purpose cores	Many SIMD-style cores grouped in SMs
Typical strength	Control flow, I/O, small tensors, data prep	Large batched linear algebra
Memory	Large RAM (64–512 GB+), lower bandwidth per core	Smaller VRAM (8–80 GB), very high bandwidth
Latency	Microseconds per complex task	Milliseconds to launch kernels; amortized over huge batches
Deep learning role	Data loading, preprocessing, orchestration	Forward pass, backward pass, optimizer math

Module 6.1 Bridge You already compute dot products and activations in forward propagation. On a GPU, those operations become thousands of parallel multiply-adds—the math is identical; only the execution model changes.

When the GPU Wins (and When It Does Not)

GPUs shine when work is massively parallel and regular: training neural networks, large-batch inference, embedding lookups at scale. CPUs win when work is serial, branchy, or tiny: parsing JSON, building feature pipelines, running sklearn on 500 rows, or serving a model where batch size is 1 and latency dominates.

Use the GPU

Training any network with meaningful batch size
Large matrix multiplies and convolutions
Multi-GPU or distributed jobs (see Module 6.2)
Batch inference (hundreds+ samples at once)

Stay on CPU

Data preprocessing and ETL
Very small models or batch size 1 latency-sensitive APIs
Development without CUDA hardware
Algorithms with heavy branching (most tree models)

PyTorch: Selecting and Using a Device

PyTorch abstracts hardware with a device object. Tensors and model parameters must live on the same device for operations to run without implicit (slow) copies.

import torch
import torch.nn as nn

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print("Using:", device)

model = nn.Sequential(
    nn.Linear(784, 256),
    nn.ReLU(),
    nn.Linear(256, 10),
).to(device)

x = torch.randn(64, 784, device=device)   # batch on GPU
y = model(x)
loss = y.sum()
loss.backward()                            # gradients stay on GPU

The pattern for every training step: load batch → .to(device) → forward → backward → optimizer step. Forgetting one tensor on CPU triggers a device-mismatch error—or worse, silent CPU fallback in some libraries.

The Hidden Cost: Host–Device Transfers

PCIe bandwidth between system RAM and VRAM is orders of magnitude slower than on-chip GPU memory. Copying a batch from CPU to GPU every iteration can erase your speedup. Best practices:

Pin CPU memory (pin_memory=True in DataLoader) for faster async transfers.
Keep the model and optimizer states on GPU for the entire training run.
Move metrics/logging to CPU only after loss.item() or .detach().cpu().

from torch.utils.data import DataLoader, TensorDataset

dataset = TensorDataset(torch.randn(10000, 784), torch.randint(0, 10, (10000,)))
loader = DataLoader(dataset, batch_size=128, pin_memory=True, num_workers=2)

for x_batch, y_batch in loader:
    x_batch = x_batch.to(device, non_blocking=True)
    y_batch = y_batch.to(device, non_blocking=True)
    # ... training step

Verifying the GPU Is Actually Busy

Code on GPU does not guarantee full utilization. Run nvidia-smi in a terminal during training: look for non-zero GPU-Util and memory usage. In PyTorch, torch.cuda.memory_allocated() reports bytes in use. Low utilization often means a small batch, CPU-bound data loading, or excessive synchronization.

Critical Mistake — “I Have a GPU So Training Is Fast”

Leaving the model on CPU while only moving input tensors, using batch_size=1, or synchronizing after every op (.item() in the inner loop) can make a GPU sit idle. Profile before buying more hardware.

Misconception — GPUs Replace CPUs

The CPU still runs Python, the DataLoader workers, checkpoint I/O, and distributed coordination. A balanced pipeline keeps the GPU fed—not starved waiting for the CPU.

Knowledge Check

Short Answer: What architectural trait makes GPUs suited to deep learning? Answer: Thousands of cores executing parallel, uniform operations (SIMD/SIMT) on large data batches.
True/False: A CPU always has more raw compute throughput than a GPU for matrix multiply. Answer: False—GPUs dominate large dense linear algebra.
Multiple Choice: Best device for parsing a CSV before training: (a) GPU, (b) CPU, (c) either. Answer: (b).
Short Answer: What does model.to(device) do? Answer: Moves all model parameters and buffers to the specified CPU or CUDA device.
Short Answer: Why use pin_memory=True? Answer: Enables faster asynchronous CPU→GPU transfers via pinned (page-locked) host memory.
True/False: Tensors on different devices can be added without error. Answer: False—PyTorch raises a device mismatch error.
Multiple Choice: Low GPU utilization during training often indicates: (a) learning rate too high, (b) data loading bottleneck, (c) too many epochs. Answer: (b).
Short Answer: What is VRAM? Answer: GPU-dedicated high-bandwidth memory holding tensors, weights, and activations during computation.
Short Answer: When is CPU inference acceptable? Answer: Small models, batch-1 latency-sensitive endpoints, or environments without CUDA.
Short Answer: What command shows live GPU memory and utilization on NVIDIA hardware? Answer: nvidia-smi.

Key Takeaways

CPUs excel at sequential, low-latency tasks; GPUs excel at massive parallel linear algebra.
Deep learning training belongs on the GPU when batch sizes and model size justify it.
Keep models, tensors, and optimizer states co-located on one device; minimize host–device copies.
Profile utilization—hardware alone does not guarantee speed.
Next: CUDA Cores — the parallel execution units inside the GPU.

Trainer’s Guide

Hands-on idea: Run the same MNIST epoch on CPU vs GPU with time.perf_counter(). Have students record batch size, epoch time, and nvidia-smi utilization.

Discussion prompt: Your API must respond in 50 ms with batch size 1. Would you deploy on GPU? What alternatives exist?

What’s Next Open CUDA Cores to see how NVIDIA’s parallel processors execute the kernels PyTorch launches behind the scenes.