CUDA — Module 18.3

How This Lesson Fits the Module & Volume

The previous lecture treated the GPU as silicon. CUDA is the NVIDIA software stack that launches kernels onto CUDA cores and tensor cores. Volume 06 explained SIMT and mixed precision; Volume 12 kernels like FlashAttention only exist as CUDA (or Triton) programs. This lesson is the compatibility and ops view: driver vs toolkit vs cuDNN vs the PyTorch wheel—the #1 reason “I bought a 4090 and torch says CUDA unavailable.”

Module 18.4’s capstone (OS, CUDA & Driver Compatibility) expands the matrix. TensorRT and ONNX Runtime CUDA EP sit on this same stack.

Learning Objectives

By the end of this lesson, students should be able to:

Separate NVIDIA driver, CUDA toolkit, cuDNN, and the PyTorch/CUDA wheel.
Map compute capability (SM version) to what kernels (FlashAttention, FP8, TensorRT) can run.
Use nvidia-smi and torch.cuda to diagnose “CUDA not available.”
Explain why a toolkit newer than the driver fails, and why a wheel CUDA version can differ from a local nvcc.
Relate CUDA streams/graphs at a high level to serving latency (without writing custom kernels).
Know when to skip CUDA entirely (Tier 1 API-only, CPU/ONNX, Apple MPS).

Definition

CUDA (Compute Unified Device Architecture) is NVIDIA’s parallel computing platform: a driver that talks to the GPU, a toolkit (nvcc, libraries), and a runtime API. Deep-learning frameworks almost never ask you to write kernels; they ship precompiled wheels linked against a specific CUDA major (11.8, 12.1, 12.4…). Your driver must be new enough for that runtime; your toolkit is only required if you compile custom CUDA/Triton/FlashAttention from source. Confusing those three layers is the root of most install threads.

Four Layers People Smash Together

Layer	What it is	You check with
NVIDIA driver	Kernel module + user libs; sets max CUDA runtime version	`nvidia-smi` (Driver Version + CUDA Version header)
CUDA toolkit	`nvcc`, headers, Nsight; compile-time only for most ML users	`nvcc --version` (may be absent and still fine)
cuDNN / cuBLAS / NCCL	Vendor libs PyTorch/TF call for conv, GEMM, all-reduce	Usually bundled inside the pip/conda wheel
Framework wheel	e.g. `torch` built for cu121	`python -c "import torch; print(torch.version.cuda, torch.cuda.is_available())"`

Compute Capability vs Marketing Name

Ampere (CC 8.0 / 8.6)

A100 (8.0), RTX 30 (8.6)
BF16 tensor cores (A100); TF32
FlashAttention 2 widely supported

Ada (CC 8.9)

RTX 40, L40/L40S
FP8 paths in TensorRT; strong infer
Watch FA / kernel pin versions

Hopper (CC 9.0)

H100 / H200
FP8 transformer engine, TMA, NVLink-4
Wrong wheel = silent CPU fallback or crash

Diagnose CUDA from Python

If is_available() is False, the wheel never attached to a driver. Common causes: CPU-only torch install, WSL without NVIDIA CUDA toolkit/WSL driver, headless server with stale driver, or a container built for a newer CUDA than the host driver. Mixed precision (autocast) only works after this check passes.

import torch

print("torch", torch.__version__, "wheel CUDA", torch.version.cuda)
print("available", torch.cuda.is_available(), "count", torch.cuda.device_count())

if not torch.cuda.is_available():
    raise SystemExit("Install a CUDA wheel + driver. CPU torch cannot see the GPU.")

dev = torch.device("cuda:0")
print("name", torch.cuda.get_device_name(0))
print("CC", ".".join(map(str, torch.cuda.get_device_capability(0))))
print("current", torch.cuda.current_device())

# Tiny FP16 GEMM — same idea as Vol. 06 mixed precision on tensor cores
a = torch.randn(1024, 1024, device=dev, dtype=torch.float16)
b = torch.randn(1024, 1024, device=dev, dtype=torch.float16)
torch.cuda.synchronize()
c = a @ b
torch.cuda.synchronize()
print("ok", c.shape, "peak alloc GiB", round(torch.cuda.max_memory_allocated() / 1024**3, 3))

# nvidia-smi counterpart: driver CUDA >= wheel CUDA runtime
# Module 18.4 OS/CUDA lecture: pin this trio in Docker (18.2) images.

Streams, Graphs, and Serving (High Level)

Use the CUDA stack when

NVIDIA GPU is the accelerator (this curriculum’s default)
You need TensorRT, FlashAttention, NCCL multi-GPU
Serving engines (vLLM, TensorRT-LLM) target CUDA kernels

Skip or replace CUDA when

Tier 1 API-only: no local kernels at all
Apple Silicon: MPS / MLX, not CUDA
Edge NPUs: TFLite / QNN / CoreML via ONNX
AMD: ROCm is analogous, not drop-in CUDA

Related Lectures

Lecture	Why it sits beside CUDA
GPU	Silicon CUDA talks to
Vol. 06 CUDA Cores	SIMT / warp model
Vol. 06 Mixed Precision	Autocast on tensor cores
Vol. 12 FlashAttention	Flagship CUDA/Triton kernel
TensorRT	Compiler that emits CUDA engines
18.4 OS & Drivers	Full compatibility matrix

Common Misconception

“I installed the CUDA toolkit, so PyTorch has GPU.” The pip/conda wheel ships its own CUDA runtime. A local nvcc does nothing if you installed CPU torch. Conversely, you can run GPU PyTorch with no toolkit if the driver is new enough. Second: the CUDA version in nvidia-smi is the maximum driver-supported runtime, not necessarily the toolkit you compiled with. Third: “CUDA = any GPU.” CUDA is NVIDIA-only; AMD is ROCm, Apple is Metal/MPS.

Knowledge Check

Short Answer: Name the four CUDA-adjacent layers ML engineers mix up. Answer: Driver, CUDA toolkit, vendor libs (cuDNN/cuBLAS/NCCL), framework wheel.
True/False: You must install nvcc to run torch.cuda.is_available() True. Answer: False—a CUDA-built wheel + compatible driver is enough.
Multiple Choice: Compute capability 9.0 is: (a) RTX 3060, (b) H100/Hopper, (c) GTX 1080. Answer: (b).
Short Answer: What does the CUDA Version line in nvidia-smi actually mean? Answer: Max CUDA runtime the installed driver can support.
True/False: FlashAttention is a CUDA/Triton kernel family, not a Python-only trick. Answer: True.
Multiple Choice: Most common cause of torch CUDA unavailable: (a) CSS cache, (b) CPU-only wheel or driver mismatch, (c) missing Redis. Answer: (b).
Short Answer: Why can wheel CUDA (e.g. 12.1) differ from nvcc 11.8 on the same box? Answer: The wheel bundles its own runtime; nvcc is a separate compile toolchain.
True/False: CUDA runs on Apple M-series GPUs. Answer: False—use MPS/MLX; CUDA is NVIDIA.
Multiple Choice: NCCL is primarily for: (a) tokenizer BPE, (b) multi-GPU collectives / all-reduce, (c) CSS animation. Answer: (b).
Short Answer: When should a team skip CUDA entirely? Answer: API-only (Tier 1), non-NVIDIA accelerators, or edge runtimes that consume ONNX/TFLite instead.

Key Takeaways

CUDA = driver + (optional) toolkit + libs + framework wheel; pin them, do not smash them.
Compute capability gates FlashAttention, FP8, and TensorRT engines.
torch.cuda + nvidia-smi diagnose 90% of “GPU not working” tickets.
Vol. 06 cores and Vol. 12 kernels assume this stack is healthy.
Continue with TensorRT—NVIDIA’s inference compiler on top of CUDA.

Trainer’s Guide

Lab: On each student machine, collect a compatibility card: driver version, nvidia-smi CUDA max, torch.version.cuda, CC, and is_available(). Deliberately install a CPU wheel in a venv and watch the check fail, then fix it.

Discussion: Should production Docker images include the full CUDA toolkit or only the runtime + cuDNN that the serving binary needs? Tie to Module 18.2 image size and 18.4 driver pinning.

Recap: CUDA is the NVIDIA software stack; match driver, wheel, and compute capability. Continue with TensorRT.