← Master Index
Vol. 18 Module 18.4 Lecture

OS, CUDA & Driver Compatibility

System Requirements — Basic to Advanced (added)

How This Lesson Fits the Module & Volume

This is the Volume 18 capstone. You can size Tiers 1–5 and still lose a week to torch.cuda.is_available() == False. Module 18.3 CUDA introduced driver vs toolkit vs wheel; this lecture is the compatibility matrix across OS, NVIDIA driver, CUDA runtime, cuDNN, PyTorch/TensorRT/ORT, and WSL2. Storage was bytes; this is boot-to-kernel truth.

Volume 19 Accuracy assumes a stack that actually runs. Pin versions in Docker (18.2) the same way you pin model dtypes (Vol. 06) and engines (Vol. 12 FlashAttention builds).

Learning Objectives

By the end of this lesson, students should be able to:

  • State the compatibility inequality: driver CUDA max ≥ wheel/runtime CUDA ≥ libraries you compile.
  • Pick an OS posture per tier: Windows/macOS Tier 1, Linux (+optional WSL2) for CUDA Tiers 2–5.
  • Diagnose common failures: CPU wheel, WDDM vs TCC, WSL driver, mismatched TensorRT, old driver on Hopper.
  • Write a one-page pin list (OS, kernel, driver, CUDA, torch, TRT) for a Docker image.
  • Know when Apple MPS/MLX or CPU ORT is the correct “compatibility” answer (skip CUDA).
  • Hand off Vol. 18 hardware to Vol. 19 evaluation without a flaky environment.
Definition

OS / CUDA / driver compatibility is the constraint that the operating system kernel module (NVIDIA driver), the CUDA user-mode runtime bundled with your framework, compiled libraries (cuDNN, NCCL, TensorRT, FlashAttention), and the GPU compute capability must agree. The driver defines the maximum CUDA runtime it will load. A newer toolkit or wheel than the driver supports fails. A GPU newer than the driver (e.g. Hopper on a stale datacenter image) fails. OS choice matters because Windows WDDM, Linux TCC/HPC drivers, and WSL2 each expose CUDA differently—and macOS has no CUDA at all.

The Inequality (memorize this)

PieceMust satisfyCheck
NVIDIA driverNew enough for GPU SKU + desired CUDA runtimenvidia-smi Driver + CUDA Version header
Framework wheelRuntime ≤ driver max; matches libc/OStorch.version.cuda, is_available()
Toolkit / nvccOnly if you compile FA/TRT pluginsnvcc --version
TensorRT / ORT CUDA EPExact CUDA + driver table from vendor docsTRT version in image tag
Compute capabilityKernels built for this SM (8.6 / 8.9 / 9.0…)get_device_capability()
OSSupported kernel + driver packageUbuntu LTS vs Win + WSL vs macOS (no CUDA)

OS Posture by Tier

Tier 1

  • Any OS: Win/macOS/Linux
  • Python + SDK; no driver matrix
  • Still pin TLS, SDK versions

Tiers 2–3 (desktop GPU)

  • Linux native = fewest surprises
  • Win + Studio/Game Ready driver works
  • WSL2: install NVIDIA WSL CUDA support
  • macOS: MPS/MLX, not CUDA

Tiers 4–5

  • Linux only in production (Ubuntu LTS / RHEL)
  • HPC datacenter drivers, not Game Ready
  • NGC / distro CUDA containers
  • Never mix random pip TRT on the host

Capstone Inventory Script

Run this on every box before you blame the model. Paste the output into the Docker tag comment.

import platform, sys, torch print("os", platform.platform()) print("python", sys.version.split()[0]) print("torch", torch.__version__, "wheel_cuda", torch.version.cuda) print("cuda_available", torch.cuda.is_available(), "count", torch.cuda.device_count()) if torch.cuda.is_available(): for i in range(torch.cuda.device_count()): p = torch.cuda.get_device_properties(i) print(f"gpu{i}", p.name, "CC", f"{p.major}.{p.minor}", "vram_GiB", round(p.total_memory / 1024**3, 2)) x = torch.randn(512, 512, device="cuda", dtype=torch.float16) torch.cuda.synchronize() print("fp16_matmul_ok", (x @ x.T).shape) else: print("HINT: CPU wheel? old driver? WSL without NVIDIA lib? macOS (no CUDA)?") print("Tier 1 is still valid — skip CUDA and use Vol. 18.1 SDKs.") # Shell companions: # nvidia-smi # driver CUDA max vs GPU recognized # nvcc --version # optional toolkit # python -c "import tensorrt" # only if TRT is in the image # Pin in Dockerfile: FROM nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04 # + torch cu124 wheel matching that major. Do not apt-get a second CUDA.

Failure Catalog (read before Slack)

Fixable mismatches

  • CPU-only torch → install CUDA wheel
  • Driver older than wheel → upgrade driver (or older wheel)
  • WSL: Windows GPU driver outdated
  • Container CUDA > host driver max
  • TRT engine built on CC 8.9 run on 8.0

Wrong tool entirely

  • macOS + CUDA expectation → MPS or Tier 1
  • AMD GPU + CUDA wheel → ROCm build
  • Tier 1 product + week of driver yak → stop, use the API
  • Game Ready on a K8s node → datacenter driver

Related Lectures (Vol. 18 map)

LectureWhy the capstone points back
18.3 CUDA / GPULayers + CC you just pinned
TensorRT / ONNXEP/engine version lock
Tier 1 … Tier 5OS posture changes with silicon
Vol. 06 CUDA CoresWhat the driver is talking to
Vol. 12 FlashAttentionOften the first compile that breaks pins
Vol. 19 AccuracyNext volume: measure quality on a stack that boots
Common Misconception

“nvidia-smi CUDA 12.6 means my PyTorch is 12.6.” That header is driver max; the wheel may be 12.1 and that is fine. Second: “Installing the full CUDA toolkit fixes CPU torch.” It does not replace a CPU wheel. Third: “Windows is unsupported for learning.” Tier 1 and many Tier 2 chat setups work; production train/serve still prefers Linux. Fourth: “Compatibility is optional if the GPU is expensive.” H100s with the wrong driver are paperweights.

Knowledge Check

  1. Short Answer: State the driver vs wheel CUDA inequality. Answer: Driver-supported CUDA max must be ≥ the framework wheel/runtime CUDA.
  2. True/False: The CUDA Version line in nvidia-smi is always the nvcc toolkit you compiled with. Answer: False—it is the driver’s maximum supported runtime.
  3. Multiple Choice: macOS CUDA story: (a) install nvidia-smi, (b) no CUDA—use MPS/MLX or Tier 1 API, (c) InfiniBand. Answer: (b).
  4. Short Answer: Why can a CUDA 12.4 container fail on a host whose nvidia-smi reports CUDA 11.8? Answer: Host driver max is 11.8; it cannot load a 12.4 runtime.
  5. True/False: A CPU-only PyTorch wheel will see the GPU after you install nvcc. Answer: False—you need a CUDA-built wheel (and a working driver).
  6. Multiple Choice: Production Tier 5 OS: (a) gaming Windows + Game Ready, (b) Linux LTS + datacenter driver + pinned NGC/CUDA image, (c) iOS. Answer: (b).
  7. Short Answer: Name one WSL2-specific failure. Answer: Outdated Windows NVIDIA driver / missing WSL CUDA user-mode libs (or GPU not passed through).
  8. True/False: TensorRT engines are portable across compute capabilities. Answer: False—rebuild per CC / TRT version.
  9. Multiple Choice: If the product is Tier 1 API-only and CUDA is broken: (a) block the course for two weeks, (b) proceed on SDKs and skip local GPU, (c) buy IB. Answer: (b).
  10. Short Answer: What Volume 19 lecture does this capstone hand off to? Answer: Vol. 19 Accuracy (module 19.1).

Key Takeaways

  • Vol. 18 capstone: pin OS + driver + CUDA runtime + wheel + CC + TRT/ORT together.
  • Driver max ≥ wheel; nvidia-smi is not nvcc; CPU torch never “sees” CUDA.
  • Tier 1 can skip the matrix; Tiers 4–5 cannot.
  • Linux + NGC images for serious GPU; macOS is MPS or API.
  • Hardware without eval is unfinished—continue to Vol. 19 Accuracy.
Trainer’s Guide

Lab (capstone): Each student fills a compatibility card for their machine (or a cloud VM): OS, driver, nvidia-smi CUDA max, torch wheel CUDA, CC, VRAM, tier they actually qualify for (1–5). Deliberately break a venv with CPU torch and repair it. Optional: write a 15-line Dockerfile FROM an official CUDA runtime that installs the matching torch extra-index wheel.

Discussion: Recap the whole volume: when should this class stay on API keys vs buy a 3060 vs rent A100s? Force storage + driver + eval (Vol. 19) into the answer, not just TFLOPS.

Recap: Volume 18 closes on OS/CUDA/driver pins so hardware tiers actually boot. Continue with Volume 19 Accuracy.