Tensor Cores — Module 6.3

How This Lesson Fits the Module

CUDA cores handle scalar FP32 math efficiently, but neural networks are dominated by matrix multiplies. NVIDIA tensor cores (introduced on Volta V100, expanded on Ampere/Hopper) are dedicated units that perform fused multiply-accumulate on small tiles—typically 4×4 or larger depending on generation—in one clock cycle class, at FP16, BF16, TF32, INT8, and more.

Tensor cores connect hardware to the precision lectures ahead: FP16, BF16, and INT8 are not just memory tricks—they are the formats tensor cores are built to chew through at peak TFLOPS.

Learning Objectives

By the end of this lesson, students should be able to:

Explain what tensor cores do versus general CUDA cores.
Describe the D = A × B + C operation tensor cores accelerate.
Identify which GPU generations include tensor cores and supported dtypes.
Enable tensor-core paths in PyTorch via mixed precision and TF32 settings.
Relate tensor core throughput to training speed and VRAM savings.

Matrix Multiply Is the Bottleneck

Linear layers, attention projections, and convolutions (via im2col) reduce to GEMM (general matrix multiply). Tensor cores execute these as warp-level matrix operations (WMMA), fusing many scalar multiply-adds into one hardware instruction path. Peak advertised TFLOPS on spec sheets often assumes tensor core FP16/BF16 throughput, not CUDA core FP32 alone.

Unit	Primary Operation	Typical Dtype	Role in DL
CUDA core	Scalar FP32 add/mul	FP32	Activations, norms, legacy paths
Tensor core	Tile matrix MAC	FP16, BF16, TF32, FP8, INT8	Linear/conv/attention GEMMs

Generations and Formats

Architecture	Example GPU	Tensor Core Notes
Volta	V100	1st gen; FP16 inputs, FP32 accumulate
Ampere	A100, RTX 3090	3rd gen; BF16, TF32, sparsity (A100)
Hopper	H100	4th gen; FP8, transformer engine
Ada	RTX 4090	Consumer; strong FP16/BF16 inference

TF32 On Ampere+, PyTorch can use TF32 for FP32 matmul internals—19-bit mantissa on tensor cores with minimal code change. Enable with torch.backends.cuda.matmul.allow_tf32 = True (often default on Ampere).

PyTorch: Routing Work to Tensor Cores

Automatic Mixed Precision (AMP) casts matmul-friendly ops to FP16 or BF16 so cuBLAS/cuDNN dispatch tensor core kernels. Master weights often stay FP32 for stability.

import torch

# Ampere+ TF32 for FP32 matmuls (training speedup, tiny numeric change)
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.allow_tf32 = True

from torch.cuda.amp import autocast

model = torch.nn.Linear(4096, 4096).cuda()
x = torch.randn(512, 4096, device="cuda")

with autocast(dtype=torch.float16):
    y = model(x)   # matmul runs on tensor cores when shapes align

print(y.dtype)     # often float16 for the matmul output path

Tensor cores require dimension alignment (multiples of 8 for FP16 on many paths). Odd layer widths may fall back to slower kernels—another reason powers-of-two hidden sizes appear in practice.

Throughput vs Numerics

Benefits

2–8× matmul speedups vs FP32 CUDA cores
Lower VRAM footprint at FP16/BF16
Higher energy efficiency per FLOP

Tradeoffs

Reduced precision → possible underflow/overflow
Loss scaling may be needed (FP16 training)
Not all ops have tensor core implementations

Misconception — Tensor Cores Run the Entire Network

Softmax, layer norm, custom losses, and small element-wise ops still run on CUDA cores or CPU paths. Speedup is concentrated in large GEMMs and convolutions.

Critical Mistake — Disabling AMP on Modern GPUs

Training large models in pure FP32 on an A100 leaves most tensor core silicon idle. Always benchmark AMP (FP16 or BF16) before assuming your baseline is optimal.

Knowledge Check

Short Answer: What operation do tensor cores specialize in? Answer: Fused matrix multiply-accumulate (D = A × B + C) on small tiles.
True/False: Tensor cores existed on the GTX 1080. Answer: False—Volta (V100) introduced them.
Multiple Choice: PyTorch AMP primarily accelerates: (a) data loading, (b) GEMM/conv in lower precision, (c) checkpoint I/O. Answer: (b).
Short Answer: What is TF32? Answer: A 19-bit format used internally for FP32 matmul on Ampere+ tensor cores.
Short Answer: Why align layer dims to multiples of 8? Answer: Tensor core kernels require aligned tile sizes for peak performance.
True/False: BF16 and FP16 both use 16 bits. Answer: True—but with different exponent/mantissa splits (covered in next lectures).
Short Answer: Name two dtypes tensor cores support on Hopper. Answer: FP8, FP16, BF16, INT8 (any two).
Multiple Choice: Spec sheet peak TFLOPS usually assumes: (a) CPU fallback, (b) tensor core low-precision GEMM, (c) single-thread FP64. Answer: (b).
Short Answer: What PyTorch context enables automatic lower-precision ops? Answer: torch.cuda.amp.autocast.
Short Answer: Do tensor cores reduce VRAM use? Answer: Yes, when weights/activations are stored in smaller dtypes.

Key Takeaways

Tensor cores are specialized matrix units that deliver most of the GPU’s deep learning FLOPS.
They operate on FP16, BF16, TF32, INT8, FP8—linking hardware to numeric precision choices.
PyTorch AMP and TF32 flags route matmuls to tensor cores with minimal code changes.
Not every op uses tensor cores; large aligned GEMMs benefit most.
Next: FP16 — the first half-precision format you will train with.

Trainer’s Guide

Hands-on idea: Benchmark one epoch with FP32 vs autocast(fp16) on the same GPU. Record wall time and max_memory_allocated().

Discussion prompt: Why do NVIDIA spec sheets list separate FP32 and tensor-core TFLOPS?

What’s Next Tensor cores need 16-bit inputs. FP16 defines what those bits mean and how to train without gradients vanishing.