BF16 — Module 6.3 | Codextroop

How This Lesson Fits the Module

FP16 saves memory but squeezes exponent bits, demanding loss scaling. BF16 (bfloat16, “Brain float”) also uses 16 bits yet keeps an 8-bit exponent like FP32 and only 7 mantissa bits. You get nearly the same dynamic range as FP32 with coarser precision—ideal for training on Ampere-and-newer tensor cores without GradScaler in many workloads.

Google Brain introduced BF16 for TPUs; NVIDIA adopted it on A100/RTX 30xx+. It is now the default mixed-precision dtype on many large-language-model training stacks.

Learning Objectives

By the end of this lesson, students should be able to:

Contrast BF16 and FP16 bit layouts and numeric tradeoffs.
Explain why BF16 often trains without loss scaling.
Enable BF16 autocast in PyTorch on supported GPUs.
Choose BF16 vs FP16 for a given hardware generation.
Recognize BF16 limitations (mantissa precision) in sensitive layers.

BF16 vs FP16: Same Size, Different Split

Format	Sign	Exponent	Mantissa	Dynamic Range	Precision
FP32	1	8	23	Wide	Highest
FP16	1	5	10	Narrow	Moderate
BF16	1	8	7	≈ FP32	Coarser

Think of BF16 as FP32 with the mantissa truncated—large and small magnitudes still representable; fine-grained fractions suffer. For gradient magnitudes spanning many orders of magnitude, that exponent match matters more than the lost mantissa bits.

Hardware Note BF16 tensor core paths require Ampere (SM 8.0) or newer on NVIDIA. Older GPUs (Pascal, Volta consumer) lack hardware BF16—use FP16 there instead.

Training with BF16 in PyTorch

import torch
from torch.cuda.amp import autocast

device = "cuda"
model = torch.nn.TransformerEncoderLayer(d_model=512, nhead=8).to(device)
optimizer = torch.optim.AdamW(model.parameters())

for src in loader:
    src = src.to(device)
    optimizer.zero_grad()

    with autocast(dtype=torch.bfloat16):
        out = model(src)
        loss = out.pow(2).mean()

    loss.backward()          # often no GradScaler needed
    optimizer.step()

PyTorch 2.x also supports torch.set_float32_matmul_precision("high") and device capability checks. On CPUs with AVX-512 BF16 (some Intel/AMD), BF16 training extends beyond NVIDIA—but this module focuses on GPU paths.

When to Pick BF16 vs FP16

Prefer BF16

Ampere/Hopper NVIDIA GPUs
Large transformer training
Want to skip GradScaler complexity
Gradients span wide magnitude ranges

Prefer FP16

Older GPUs without BF16 tensor cores
Inference stacks expecting IEEE half
Need slightly finer mantissa (some vision tasks)
Legacy ONNX/TensorRT FP16 pipelines

Misconception — BF16 Is Always Better Than FP16

BF16 sacrifices mantissa precision. Tasks sensitive to tiny weight updates or very small activations may still prefer FP32 accumulation or FP16 with careful scaling. Benchmark on your model.

Critical Mistake — BF16 on Unsupported Hardware

Requesting autocast(dtype=torch.bfloat16) on a GTX 1080 falls back or errors. Check torch.cuda.get_device_capability() — major version ≥ 8 for native BF16 tensor cores.

BF16 and Memory

Like FP16, BF16 halves tensor storage versus FP32. Combined with VRAM strategies, BF16 enables larger batch sizes on the same card. Optimizer states may still be FP32 depending on framework defaults (e.g., fused AdamW keeps master weights FP32).

Knowledge Check

Short Answer: What does “BF” stand for? Answer: Brain (Google Brain float format).
Short Answer: How does BF16 exponent compare to FP32? Answer: Same 8-bit exponent—similar dynamic range.
True/False: BF16 and FP16 use the same bit layout. Answer: False—different exponent/mantissa split.
Multiple Choice: BF16 training often skips: (a) optimizer, (b) GradScaler, (c) backward. Answer: (b).
Short Answer: Minimum NVIDIA architecture for BF16 tensor cores? Answer: Ampere (compute capability 8.0).
Short Answer: What BF16 sacrifices vs FP32? Answer: Mantissa precision (only 7 mantissa bits).
True/False: BF16 uses 32 bits per value. Answer: False—16 bits.
Multiple Choice: Wider dynamic range at 16 bits: (a) FP16, (b) BF16, (c) INT8. Answer: (b).
Short Answer: PyTorch autocast line for BF16? Answer: with autocast(dtype=torch.bfloat16):
Short Answer: Why do LLM trainers default to BF16 on A100/H100? Answer: FP32-like range without loss scaling, fast tensor cores, half memory.

Key Takeaways

BF16 = FP32 exponent + truncated mantissa in 16 bits.
Dynamic range like FP32; precision coarser than FP16 in fractional bits.
Preferred training dtype on Ampere+ when hardware supports it.
Often trains without GradScaler; still validate loss curves against FP32.
Next: INT8 — integer quantization for inference speed.

Trainer’s Guide

Hands-on idea: On an Ampere GPU, train the same model in FP16 (with scaler) vs BF16 (without). Compare loss curves and step time.

Discussion prompt: Why did Google optimize BF16 for TPUs while NVIDIA added it to tensor cores?

What’s Next BF16 is still floating point. INT8 pushes further—8-bit integers for deployment-grade inference compression.