Tier 5 — Production / Training Cluster (A100/H100 clusters, distributed training, NVLink/InfiniBand) — Module 18.4

How This Lesson Fits the Module & Volume

Tier 4 is a desk or a single cloud node. Tier 5 is a fabric: A100/H100 (and H200/B200) pods, NVLink / NVSwitch inside the node, InfiniBand (or RoCE) between nodes, and distributed training (DDP, FSDP/ZeRO, tensor/pipeline parallel). Volume 12 serving ideas scale too: vLLM/TensorRT-LLM on multi-GPU with paged KV and FlashAttention. Most teams should rent this tier; almost nobody should build a closet InfiniBand mesh “to learn.”

If a hosted API (Tier 1) or a QLoRA workstation (Tier 4) meets the SLO, staying lower is not a failure—it is cost control.

Learning Objectives

By the end of this lesson, students should be able to:

Describe a DGX-class node: 8× GPU, NVLink/NVSwitch, local NVMe, IB NICs.
Distinguish NVLink (intra-node) from InfiniBand (inter-node) and why both matter.
Place DDP vs FSDP/ZeRO vs tensor/pipeline parallel on the right problem.
Know what Tier 5 is for: 70B+ FP16/BF16 train, multi-node pretrain, high-QPS multi-GPU serve.
List ops realities: NCCL, topology, CUDA/driver pinning, storage bandwidth (next lecture).
Decide rent vs buy vs stay on Tier 1/4 with a TCO argument.

Definition

Tier 5 (Production / Training Cluster) is multi-GPU, usually multi-node NVIDIA datacenter hardware (A100/H100-class) connected by a high-bandwidth, low-latency fabric. NVLink (and NVSwitch) provides GPU-to-GPU bandwidth inside a server far above PCIe. InfiniBand (or equivalent RDMA Ethernet) provides GPU-direct-ish communication across servers so NCCL all-reduces and tensor-parallel shards do not stall on commodity 10 GbE. The software is PyTorch DDP/FSDP, Megatron-style TP/PP, DeepSpeed ZeRO, and production inference engines—not Ollama on a 3060.

Node vs Fabric

Layer	Typical Tier 5	If you only have PCIe desktop
GPU	8× A100 80 GB or 8× H100 80/94 GB (H200 more HBM)	1–2 consumer cards (Tier 3–4)
Intra-node	NVLink 3/4 + NVSwitch; hundreds GB/s GPU–GPU	PCIe 4/5 x16; TP 70B will crawl
Inter-node	HDR/NDR InfiniBand, rail-optimized	Office Ethernet; multi-node train is a science project
Storage	Parallel FS / NVMe + object store (next lecture)	One SSD; checkpoint IO becomes the bottleneck
Software	NCCL, CUDA HPC drivers, Slurm/K8s, NGC images	pip + hope

Parallelism Menu (Vol. 06, production reading)

DDP / data parallel

Replicate full model; split batch
Needs model to fit one GPU
All-reduce grads (IB sensitive at scale)

ZeRO / FSDP

Shard params/grads/opt states
Fits larger models per node
More communication; needs fat fabric

Tensor / pipeline parallel

Split layers or matmuls across GPUs
70B+ FP16 train/serve
Wants NVLink inside the node

NCCL Sanity (you are on a cluster now)

Before a week-long pretrain, prove topology. Mixed PCIe + IB, wrong NCCL P2P, or a single slow NIC shows up as “H100s are slow.” Serving: tensor-parallel vLLM/TRT-LLM still budgets KV per rank.

import os, torch
import torch.distributed as dist

# Typical launch: torchrun --nnodes=2 --nproc_per_node=8 --rdzv_backend=c10d \
#   --rdzv_endpoint=$MASTER_ADDR:29500 train.py

def main():
    dist.init_process_group(backend="nccl")
    local = int(os.environ["LOCAL_RANK"])
    torch.cuda.set_device(local)
    r, w = dist.get_rank(), dist.get_world_size()
    print(f"rank {r}/{w} gpu {torch.cuda.get_device_name(local)}", flush=True)

    t = torch.ones(1, device=f"cuda:{local}") * r
    dist.all_reduce(t, op=dist.ReduceOp.SUM)  # tiny NCCL canary
    if r == 0:
        print("all_reduce sum", int(t.item()), "expected", sum(range(w)))

    # Real jobs: FSDP/Megatron/DeepSpeed configs — not this file.
    # Env often set by scheduler: NCCL_IB_HCA, NCCL_P2P_LEVEL, CUDA_VISIBLE_DEVICES
    dist.destroy_process_group()

if __name__ == "__main__":
    main()
# nvidia-smi topo -m  → confirm NVLink vs PCIe between GPUs
# If all_reduce is slow, fix fabric before buying more H100s.

Rent vs Build vs Avoid

Use Tier 5 when

70B+ FP16/BF16/FP8 train or TP serve
Multi-node pretrain / continued pretrain
High-QPS production with SLO + multi-GPU engines
You have platform eng (NCCL, storage, K8s)

Do not build a closet cluster when

Tier 1 API quality/TCO still wins
QLoRA on one 80 GB box (Tier 4) solves FT
No IB, no NVSwitch, “8×4090 on USB” fantasies
Team cannot operate CUDA/NCCL/storage yet

Related Lectures

Lecture	Why it sits beside Tier 5
Vol. 06 Distributed Training	DDP/FSDP concepts
18.3 CUDA / GPU	Datacenter SKUs + NCCL
Vol. 12 KV / FA	Multi-GPU serve still memory-bound
18.2 Kubernetes	How clusters are scheduled
Storage & Bandwidth	Checkpoints will saturate disks next
Tier 1	The alternative you should still cost out

Common Misconception

“Eight 4090s in a miner chassis is a mini-DGX.” Without NVLink/NVSwitch and a real IB fabric, tensor parallel and multi-node all-reduce do not behave like H100 pods. Second: “NCCL is just PyTorch DDP.” DDP calls NCCL; bad topology makes NCCL the bottleneck. Third: “Tier 5 replaces quantization.” FP8/INT8 still pay rent on H100 serve; Vol. 06 dtypes did not retire.

Knowledge Check

Short Answer: Name the two fabrics in the Tier 5 title and their usual scope. Answer: NVLink (intra-node GPU–GPU) and InfiniBand (inter-node RDMA).
True/False: DDP shards 70B weights across GPUs by default. Answer: False—DDP replicates; use FSDP/ZeRO or TP/PP to shard.
Multiple Choice: Best interconnect for 8 GPUs inside one DGX-class node: (a) USB4, (b) NVLink/NVSwitch, (c) Wi-Fi 6. Answer: (b).
Short Answer: Why does 10 GbE office Ethernet fail multi-node H100 train? Answer: All-reduce/TP traffic needs IB-class bandwidth/latency; 10 GbE stalls NCCL.
True/False: Most product teams should rent Tier 5 rather than build InfiniBand in a closet. Answer: True.
Multiple Choice: Tiny NCCL canary in PyTorch: (a) dist.all_reduce on CUDA tensors, (b) CSS minify, (c) Redis PING. Answer: (a).
Short Answer: When is FSDP/ZeRO preferred over DDP? Answer: When the full model + Adam does not fit on one GPU and you must shard states.
True/False: Multi-GPU vLLM still must budget KV cache (Vol. 12). Answer: True.
Multiple Choice: QLoRA 13B weekly on private data, one node: (a) mandatory 64-GPU IB pod, (b) often Tier 4 80 GB, (c) MCU. Answer: (b).
Short Answer: What lecture follows because checkpoints will dwarf GPU RAM? Answer: Storage & Bandwidth Planning.

Key Takeaways

Tier 5 = A100/H100-class pods + NVLink + InfiniBand + distributed training/serve.
DDP replicates; FSDP/ZeRO/TP/PP shard; fabric quality is part of the algorithm.
Rent by default; closet 4090 meshes are not DGX.
Still cost out Tier 1 and Tier 4 before you NCCL.
Continue with Storage & Bandwidth Planning.

Trainer’s Guide

Lab: Diagram a 2-node 8×H100 job: where TP vs DP vs PP live, where NVLink vs IB runs, and what fails if IB is down. If a multi-GPU box exists, run the all_reduce canary and nvidia-smi topo -m.

Discussion: A board wants “our own ChatGPT cluster.” Force TCO vs API, data residency, and platform-engineering headcount—not just GPU list price.

Recap: Tier 5 is distributed A100/H100 training and serve on NVLink + InfiniBand. Continue with Storage & Bandwidth Planning.