CUDA Cores

How This Lesson Fits the Module

After CPU vs GPU, you know why GPUs accelerate deep learning. CUDA cores are the “how”—NVIDIA’s name for the scalar floating-point processors inside each Streaming Multiprocessor (SM) that execute the thousands of threads launched by a CUDA kernel or a PyTorch operation.

CUDA cores handle general FP32 (and FP64 on some lines) math. Later lectures introduce tensor cores for specialized matrix math and numeric formats like FP16. Think of CUDA cores as the GPU’s workhorse infantry; tensor cores are the specialized artillery.

Learning Objectives

By the end of this lesson, students should be able to:

Define a CUDA core and its role inside a Streaming Multiprocessor.
Explain the SIMT (Single Instruction, Multiple Thread) execution model.
Relate PyTorch GPU ops to underlying CUDA kernel launches.
Distinguish CUDA cores from tensor cores and from CPU cores.
Read a GPU spec sheet: SM count, CUDA cores per SM, clock speed.

What Is a CUDA Core?

A CUDA core is an NVIDIA marketing and architecture term for a single FP32 ALU (arithmetic logic unit) in the GPU shader pipeline. Modern GPUs contain many Streaming Multiprocessors (SMs), each bundling dozens of CUDA cores plus shared memory, schedulers, and load/store units. An RTX 4090 has 128 SMs × 128 CUDA cores/SM = 16,384 CUDA cores—but they are not independent like CPU cores; they execute in lockstep groups called warps (32 threads).

Concept	CPU Core	CUDA Core (within SM)
Independence	Runs its own instruction stream	Executes same instruction as warp peers (SIMT)
Branching	Efficient complex control flow	Divergent branches serialize within a warp
Primary workload	General-purpose threads	Parallel FP32 math on arrays
Context switch	Expensive	Hardware-scheduled warps hide latency

From PyTorch to CUDA Kernels

When you call torch.matmul(a, b) on a CUDA tensor, PyTorch dispatches to cuBLAS (or similar), which launches CUDA kernels that fan work across SMs and CUDA cores. You rarely write kernels directly in introductory deep learning—but every layer uses them.

import torch

a = torch.randn(4096, 4096, device="cuda")
b = torch.randn(4096, 4096, device="cuda")

# cuBLAS GEMM — saturates CUDA cores across SMs
c = torch.matmul(a, b)
print(c.shape, c.device)

For custom element-wise work, you can write a minimal CUDA kernel (via PyTorch C++ extension or torch.cuda APIs). The pattern: define a kernel, specify a grid of thread blocks, launch on the GPU stream.

# Conceptual CUDA C++ kernel (not run in browser)
__global__ void add_vectors(const float* a, const float* b, float* out, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) out[i] = a[i] + b[i];
}
// Launch: <<>>(a, b, out, n);

Execution Model One kernel launch → many thread blocks → warps of 32 threads → each thread may map to one element. Occupancy (active warps per SM) determines how well CUDA cores stay busy while waiting on memory.

Reading a GPU Spec Sheet

Spec	Example (A100 80GB)	What It Tells You
CUDA cores	6,912	Peak parallel FP32 throughput (theoretical)
SM count	108	Independent execution clusters
Boost clock	~1.4 GHz	Higher clock → more ops/sec per core
Tensor cores	432 (3rd gen)	Separate units for matrix multiply—see next lectures

CUDA Cores vs Tensor Cores

General FP32 (and limited FP64) ops
Element-wise activations, reductions, legacy GEMM paths
Present on all CUDA-capable NVIDIA GPUs

Tensor Cores

Specialized D = A × B + C matrix ops
FP16, BF16, INT8, TF32 mixed precision
Volta (V100) and newer datacenter/consumer lines

Misconception — More CUDA Cores Always Means Faster Training

Memory bandwidth, tensor core availability, batch size, and software stack matter as much as core count. A well-fed older GPU can beat an underutilized newer one.

Critical Mistake — Warp Divergence in Custom Kernels

If threads in the same warp take different if/else branches, the hardware serializes both paths. Keep branch conditions uniform across warp lanes when writing CUDA by hand.

Knowledge Check

Short Answer: What is a CUDA core? Answer: An FP32 ALU in an NVIDIA SM that executes one thread’s scalar math per cycle in the SIMT model.
Short Answer: What is a warp? Answer: A group of 32 threads executed in lockstep on the same instruction.
True/False: Each CUDA core runs a fully independent OS thread like a CPU core. Answer: False—SIMT groups share instruction streams.
Multiple Choice: torch.matmul on CUDA tensors primarily uses: (a) CPU BLAS, (b) cuBLAS/CUDA kernels, (c) Python loops. Answer: (b).
Short Answer: What is an SM? Answer: Streaming Multiprocessor—a cluster of CUDA cores, schedulers, and on-chip memory.
Short Answer: What is SIMT? Answer: Single Instruction, Multiple Threads—one instruction broadcast to many threads.
True/False: Tensor cores replace CUDA cores entirely on modern GPUs. Answer: False—both coexist; frameworks route ops to the best unit.
Short Answer: What hurts warp efficiency in custom kernels? Answer: Branch divergence within a warp.
Multiple Choice: Occupancy refers to: (a) VRAM usage, (b) active warps per SM, (c) batch size. Answer: (b).
Short Answer: Why don’t deep learning practitioners write CUDA for every layer? Answer: Optimized libraries (cuBLAS, cuDNN) already provide highly tuned kernels.

Key Takeaways

CUDA cores are the parallel FP32 units inside each SM that execute GPU kernels.
Threads are grouped into warps (32); SIMT execution favors uniform, data-parallel code.
PyTorch GPU operations launch CUDA kernels under the hood—you benefit without writing C++.
CUDA cores handle general math; tensor cores accelerate matrix multiply at lower precision.
Next: VRAM — the memory pool those cores read and write.

Trainer’s Guide

Hands-on idea: Use torch.cuda.get_device_properties(0) to print SM count and total cores. Compare two GPUs in the classroom.

Discussion prompt: Why does doubling CUDA core count not always double training throughput?

What’s Next CUDA cores compute fast only when data is local. VRAM explains how much fits on the card and what happens when it does not.