← Master Index
Vol. 18 Module 18.4 Lecture

Storage & Bandwidth Planning

System Requirements — Basic to Advanced (added)

How This Lesson Fits the Module & Volume

Tiers 1–5 sized compute. GPUs starve without disks and pipes. Storage & bandwidth planning covers checkpoints, datasets, Hugging Face hub pulls, object-store egress, NVMe vs HDD, and why a Tier 5 node with slow NFS trains like a Tier 2 hobby box. Volume 06 checkpoint lectures and Volume 12 long-context serving both assume tokens and weights can move faster than the step time.

The next (capstone) lecture pins OS/CUDA/drivers. Together they finish Vol. 18’s hardware playbook before Vol. 19 Accuracy asks whether any of this was worth it.

Learning Objectives

By the end of this lesson, students should be able to:

  • Estimate disk for weights, tokenizer artifacts, datasets, logs, and N checkpoint revisions.
  • Plan per-tier storage: Tier 1 (almost none) through Tier 5 (parallel FS + object store).
  • Separate disk throughput (NVMe) from network bandwidth (WAN egress, IB, Ethernet).
  • Avoid dataloader and checkpoint IO as hidden step-time killers.
  • Account for cloud egress + Hub rate limits in TCO next to GPU rent.
  • Write a simple capacity spreadsheet the ops team can actually use.
Definition

Storage planning sizes persistent bytes: model repos, sharded datasets, experiment checkpoints, indexes (RAG), and logs. Bandwidth planning sizes bytes/second: NVMe read for dataloaders, PCIe for CPU↔GPU copies, LAN/IB for multi-node, and WAN for Hub/S3/API. A GPU with empty PCIe lanes or a 100 MB/s NFS checkpoint path is not a Tier 5 machine—it is a very expensive wait state. Quantized weights (Module 18.3) shrink disk as well as VRAM; they do not shrink raw datasets.

What Eats Disk (order-of-magnitude)

ArtifactRough sizeWho cares
7B FP16 / INT4 weights~14 GB / ~4 GBTiers 2–3 local cache
70B FP16 / INT4~140 GB / ~40 GBTier 4–5; multi-revision ×N
HF hub snapshot (full repo)Weights + tokenizer + extra shardsEveryone who “just clone”
SFT dataset (JSONL text)1–100+ GB raw; more tokenizedTier 4 dataloaders want local NVMe
Pretrain tokens (web scale)TBs–PBsTier 5 parallel FS / object store
Adam checkpoint (full FT)~2–4× weight sizeDo not keep 50 full FT ckpts on one SSD
LoRA adapter only10–500 MBKeep many; cheap
RAG index + embeddingsDocs × dim × 4 bytes (+ overhead)Often larger than the LLM on Tier 1–2

Per-Tier Playbook

Tiers 1–2

  • T1: almost no model disk; log + cache only
  • T2: 1 TB NVMe covers Q4 7Bs + a small corpus
  • Watch Hub re-downloads on flaky Wi-Fi
  • API egress = token/audio/image bytes, not IB

Tiers 3–4

  • 2–8 TB NVMe: 13B/70B shards + ckpts + ds
  • Keep LoRAs; prune full FT ckpts
  • Dataloader: local SSD, not spinning HDD
  • 64–256 GB host RAM for page cache helps

Tier 5

  • Node-local NVMe scratch + shared FS
  • Object store (S3/GCS/MinIO) for durable ckpts
  • IB for grads; still need fat storage net
  • Checkpoint every N min without stalling GPUs

A Tiny Capacity Spreadsheet in Code

Plan revisions explicitly. “We have 2 TB” dies when someone saves 20 full 70B Adam dumps.

def gb(n): # n in bytes → GiB return round(n / 1024**3, 2) params = 70e9 bytes_per = {"fp16": 2, "int4": 0.55, "adam_full": 8} # adam_full ≈ rough opt+grad+w revisions = 5 dataset_gb = 800 lora_mb = 80 lora_versions = 40 weights = params * bytes_per["fp16"] ckpts = revisions * params * bytes_per["adam_full"] loras = lora_versions * lora_mb * 1024**2 print("70B fp16 weights GiB", gb(weights)) print("5 full Adam ckpts GiB", gb(ckpts)) print("40 LoRAs GiB", gb(loras)) print("plus dataset GiB", dataset_gb) print("TOTAL GiB", gb(weights + ckpts + loras) + dataset_gb) # Bandwidth sketch: checkpoint 140 GB / 20 s write → 7 GB/s needed (NVMe RAID / parallel FS) # HDD @ 200 MB/s → 12+ minutes per dump → GPUs idle. That is not Tier 5. # WAN: Hub pull 140 GB on a 100 Mbps link → hours; cache on-prem or use a region close to GPUs. # Tier 1 TCO cousin: image/audio API payloads can dwarf text tokens — meter them.

IO That Looks Like a Model Bug

Do this

  • NVMe for active ds + scratch ckpts
  • Object store for durable history
  • Save LoRA deltas, not 20 full bases
  • Prefetch dataloaders; pin memory
  • Colocate Hub cache with the GPU region

Stop doing this

  • Training off a USB HDD or home NAS at 40 MB/s
  • Re-downloading 70B every job
  • Checkpointing full Adam every step
  • Ignoring cloud egress until the bill
  • Putting RAG indexes on the same tiny OS disk

Related Lectures

LectureWhy it sits beside storage
Vol. 06 CheckpointsWhat you are persisting
18.3 QuantizationINT4 shrinks weight disk too
Vol. 12 LoRATiny artifacts vs full FT dumps
Tier 5IB ≠ storage network automatically
18.2 DockerLayer cache vs model cache
OS, CUDA & DriversNext: the other silent outage class
Common Misconception

“VRAM size is the only capacity number.” Datasets and checkpoints are often 10–100× VRAM. Second: “InfiniBand solves checkpoint speed.” IB moves grads; dumps still need a storage fabric or local NVMe. Third: “Tier 1 has zero bandwidth concerns.” API image/video payloads and Hub downloads still hit WAN and laptop disks.

Knowledge Check

  1. Short Answer: Name three artifact classes you must size besides GPU VRAM. Answer: Weights/repos, datasets, checkpoints (also RAG indexes, logs).
  2. True/False: A 70B FP16 checkpoint is roughly the same size as a LoRA adapter. Answer: False—~140 GB vs tens–hundreds of MB.
  3. Multiple Choice: Active SFT dataloader disk: (a) USB HDD, (b) local NVMe, (c) optical disc. Answer: (b).
  4. Short Answer: Why do 20 full Adam 70B dumps blow a 2 TB SSD? Answer: Each dump is multiple× weight size; 20 revisions are many TB.
  5. True/False: InfiniBand automatically makes NFS checkpoint writes fast. Answer: False—storage path is separate unless designed that way.
  6. Multiple Choice: Tier 1 storage footprint: (a) PB parallel FS required, (b) tiny—app + logs + optional cache, (c) 8×H100 scratch only. Answer: (b).
  7. Short Answer: How does quantization help storage as well as VRAM? Answer: Fewer bytes/param on disk for weight shards (INT4 vs FP16).
  8. True/False: Cloud egress and Hub pull time belong in TCO next to GPU hours. Answer: True.
  9. Multiple Choice: Best durable home for many full ckpts: (a) the OS disk, (b) object store + lifecycle rules, (c) RAM disk only. Answer: (b).
  10. Short Answer: What Vol. 18 lecture comes next as the compatibility capstone? Answer: OS, CUDA & Driver Compatibility.

Key Takeaways

  • Plan weights × revisions, datasets, LoRAs vs full FT, and RAG indexes—not just VRAM.
  • NVMe for hot IO; object store for history; IB is not a substitute for disks.
  • Per-tier: T1 almost none → T5 parallel FS + scratch.
  • WAN egress and Hub cache are part of hardware TCO.
  • Continue with OS, CUDA & Driver Compatibility.
Trainer’s Guide

Lab: Run the capacity snippet for a fictional lab (two 13B FP16, one 70B INT4, 15 LoRAs, 400 GB dataset, 8 full FT ckpts). Students pick disk SKUs per tier and estimate Hub pull time at 200 Mbps.

Discussion: Checkpoint every step vs every epoch vs every 30 minutes—how does that interact with GPU utilization and disaster recovery?

Recap: Storage and bandwidth are first-class hardware. Continue with OS, CUDA & Driver Compatibility.