GPU — Module 18.3 | Codextroop

How This Lesson Fits the Module & Volume

Module 18.2 Observability taught you to see a live service. Module 18.3 asks the hardware question: can this model even fit on the box you bought? Volume 06 already defined CUDA cores, VRAM, and dtypes (FP16 / BF16 / INT8). This lecture is the production reading of those specs—consumer vs datacenter cards, what actually occupies VRAM at serve time, and when a GPU is the wrong purchase.

Volume 12 inference work (KV cache, FlashAttention) is why two GPUs with the same GB rating feel different: decode is memory-bandwidth bound, not just “more CUDA cores.” Module 18.4 then turns this into a sizing playbook from API-only laptops to H100 clusters.

Learning Objectives

By the end of this lesson, students should be able to:

Read a GPU spec sheet for SM/CUDA cores, tensor cores, VRAM capacity, and memory bandwidth.
Estimate weight + activation + KV-cache VRAM for a transformer at a given dtype and context.
Contrast consumer (RTX 30/40) vs workstation (A6000) vs datacenter (A100/H100) cards.
Use torch.cuda to inventory devices, compute capability, and allocated memory.
Explain why FlashAttention and paged KV change effective capacity more than a clock bump.
Decide when to stay API-only instead of buying a GPU (preview of Module 18.4 Tier 1).

Definition

A GPU (graphics processing unit) is a massively parallel accelerator whose usable deep-learning capacity is the product of three things: (1) VRAM—the dedicated high-bandwidth pool that must hold weights, activations, optimizer state (train), and the KV cache (decode); (2) compute—CUDA cores for general math plus tensor cores for matrix multiply-accumulate in FP16/BF16/INT8/FP8; (3) memory bandwidth—GB/s from VRAM to SMs, which dominates autoregressive decode. Buying “a 4090” without naming dtype, context length, and concurrent users is not a hardware plan.

What Occupies VRAM in Production

Volume 06 treated VRAM as weights + activations + optimizer. Serving adds the KV cache: for every layer, every token in the live context, you store K and V. Long context and multi-user batch explode this term. FlashAttention does not shrink the KV store, but it slashes activation scratch during attention, which is why the same 24 GB card can run a larger batch after an engine upgrade.

Resident	Rough size	Who pays
Weights	~2 bytes/param FP16; ~0.5 bytes/param INT4	Every request (shared)
Activations / scratch	Batch × seq × hidden; FlashAttention reduces peak	Prefill more than decode
KV cache	2 × layers × seq × kv_heads × head_dim × dtype × batch	Each concurrent sequence
Optimizer (Adam)	~8–16 bytes/param extra vs inference	Training / full FT only
CUDA context + fragmentation	0.5–2+ GB tax; leave headroom	Always

Consumer vs Workstation vs Datacenter

Consumer (3060 / 4060 / 4090)

8–24 GB GDDR; great $/perf for hobby + Tier 2/3 infer
No ECC; driver/WDDM quirks on Windows
NVLink absent on 40-series desktop
Fine for INT4 7B; 4090 can FP16 13B–small-34B with care

Workstation (A6000 / L40S)

48 GB (A6000) class; ECC; multi-GPU PCIe/NVLink on some SKUs
Quiet enough for an office; still not InfiniBand
LoRA / QLoRA workstation (Tier 4 preview)

Datacenter (A100 / H100 / H200)

40–141 GB HBM; NVLink + NVSwitch fabrics
MIG, FP8 tensor cores (Hopper), cluster interconnect
Rent before you buy; this is Tier 5

Inventory the Box with torch.cuda

Never trust a shop listing. Query compute capability (CC), VRAM, and a tiny matmul so you know the driver + PyTorch wheel actually talk to the card. CC 8.6 (Ampere consumer) vs 8.9 (Ada) vs 9.0 (Hopper) gates FlashAttention builds, FP8, and some TensorRT engines.

import torch

print("cuda available:", torch.cuda.is_available())
print("device count:", torch.cuda.device_count())
print("built CUDA:", torch.version.cuda)

if torch.cuda.is_available():
    i = torch.cuda.current_device()
    p = torch.cuda.get_device_properties(i)
    print(p.name, "CC", f"{p.major}.{p.minor}", "SMs", p.multi_processor_count)
    print("total VRAM GiB:", round(p.total_memory / 1024**3, 2))
    x = torch.randn(4096, 4096, device="cuda", dtype=torch.float16)
    y = x @ x.T
    torch.cuda.synchronize()
    print("allocated GiB:", round(torch.cuda.memory_allocated() / 1024**3, 3))
    print("reserved  GiB:", round(torch.cuda.memory_reserved() / 1024**3, 3))
# OOM? lower dtype (Vol. 06 FP16/INT8) or quantize (this module) before buying RAM.

When a GPU Is the Wrong Buy

Buy / rent a GPU when

You need local weights (privacy, air-gap, latency < API RTT)
Token volume makes API TCO worse than a 24–80 GB box
You will LoRA/QLoRA or run continuous batching locally
You already understand KV + dtype budgets (Vol. 06 + 12)

Stay API-only when

Laptop + internet + key is enough (Module 18.4 Tier 1)
Peak load is bursty; idle silicon is expensive
You need frontier models you cannot host
Team cannot own drivers, CUDA, and observability yet

Related Lectures

Lecture	Why it sits beside GPU
Vol. 06 CUDA Cores / Tensor Cores	What the silicon actually is
Vol. 06 VRAM	Capacity math; OOM taxonomy
FP16 / BF16 / INT8	Dtype × bytes/param
Vol. 12 KV Cache	Decode memory that eats VRAM
FlashAttention	I/O-aware attention; more batch in same GB
18.4 Tier 1	Explicit “no GPU” playbook

Common Misconception

“More CUDA cores means a bigger model fits.” Fit is almost entirely VRAM + fragmentation + KV, not core count. A 4090 (24 GB, huge CUDA count) can lose to a 48 GB A6000 on 34B FP16 even if the 4090 wins synthetic TFLOPS. Second misconception: “24 GB is 24 GB of model.” CUDA context, KV, and scratch steal several GB; plan ~80% usable. Third: “I need a GPU to learn GenAI.” Module 18.1 SDKs + Tier 1 API-only are a complete path until you outgrow them.

Knowledge Check

Short Answer: Name the three GPU resources that jointly set LLM serving capacity. Answer: VRAM capacity, compute (CUDA/tensor cores), and memory bandwidth.
True/False: CUDA core count is the main determinant of whether a 13B FP16 model fits. Answer: False—VRAM (plus KV/scratch) determines fit.
Multiple Choice: Decode-time VRAM growth is dominated by: (a) optimizer states, (b) KV cache per live token/user, (c) CSS themes. Answer: (b).
Short Answer: Why does FlashAttention help a 24 GB card feel larger? Answer: It reduces attention activation/scratch I/O so peak memory and bandwidth waste drop; KV itself still remains.
True/False: An RTX 4090 and an A6000 with equal advertised TFLOPS have equal model-fit. Answer: False—24 GB vs 48 GB VRAM (and ECC/NVLink) dominate fit.
Multiple Choice: torch.cuda.get_device_properties reports: (a) only RAM price, (b) name, compute capability, SM count, total memory, (c) Git SHA. Answer: (b).
Short Answer: Rough FP16 weight size for a 7B model? Answer: ~14 GB (7e9 × 2 bytes), plus KV/scratch/CUDA tax.
True/False: INT4 quantization can make a 7B model fit an 8–12 GB consumer GPU. Answer: True (typical ~4–6 GB weights + headroom).
Multiple Choice: Stay API-only when: (a) you must air-gap weights, (b) bursty load + no ops budget for drivers, (c) you already own an H100 cluster idle. Answer: (b).
Short Answer: What Vol. 12 structure must you budget besides weights? Answer: The KV cache (and usually paged/block layout under a serving engine).

Key Takeaways

GPUs serve GenAI through VRAM + tensor-core compute + bandwidth—not marketing TFLOPS alone.
Weights, KV cache, scratch, and CUDA tax share one pool; dtype and context set the budget.
Vol. 06 silicon + Vol. 12 KV/FlashAttention are the theory; this lecture is the shop-floor reading.
Inventory with torch.cuda; do not buy from a listing screenshot.
Continue with CUDA—the software stack that actually talks to the card.

Trainer’s Guide

Lab: Run the torch.cuda inventory on whatever machine students have (CPU-only is a valid result). Estimate FP16 vs INT4 weight GB for 7B and 13B, then add a toy KV term: 32 layers × 4096 seq × 8 kv heads × 128 dim × 2 (K+V) × 2 bytes.

Discussion: A startup wants a 70B “local ChatGPT.” Do they buy 2×4090, rent A100s, or stay on the API? Force them to name dtype, context, and concurrent users before naming SKUs.

Recap: A GPU is a VRAM + tensor-core + bandwidth budget. Read Vol. 06 specs and Vol. 12 KV/FlashAttention before you purchase. Continue with CUDA.