TensorRT — Module 18.3

How This Lesson Fits the Module & Volume

CUDA gives you kernels. TensorRT is NVIDIA’s inference compiler: it takes a trained graph (often via ONNX), fuses layers, picks tactic kernels for your exact GPU + dtype, and emits a serialized engine. Volume 06 INT8 / FP16 and Volume 12 FlashAttention-style I/O awareness show up here as compiler passes, not as handwritten CUDA.

For LLMs, TensorRT-LLM (and engines inside Triton Inference Server) is the production cousin of “just run Hugging Face generate().” You pay build time and shape constraints; you gain latency and throughput on NVIDIA silicon. Next lecture (ONNX) is the portable IR TensorRT usually consumes.

Learning Objectives

By the end of this lesson, students should be able to:

Define TensorRT as an NVIDIA inference compiler that produces hardware-specific engines.
List the main optimizations: layer fusion, kernel auto-tune, precision calibration, memory reuse.
Contrast PyTorch eager / torch.compile vs TensorRT vs TensorRT-LLM.
Explain ONNX → parse → build → serialize → infer, and why engines are not portable across GPU SKUs.
State when TensorRT is worth it vs staying in a CUDA serving engine (vLLM) or API-only.
Relate INT8 calibration to Vol. 06 quantization theory without treating it as magic accuracy.

Definition

TensorRT is NVIDIA’s high-performance deep-learning inference SDK. Given a network (ONNX, or a framework parser) and a target GPU, it builds an optimized engine: a binary plan of fused kernels, chosen tactics, and workspace layout for fixed or optimized dynamic shapes. The engine is not a checkpoint. Rebuild when GPU architecture, TensorRT version, or precision changes. TensorRT-LLM specializes this pipeline for decoder LLMs (KV cache, inflight batching, FP8 on Hopper)—the Vol. 12 serving ideas compiled onto NVIDIA hardware.

What the Compiler Actually Does

Pass	Effect	Curriculum link
Layer fusion	Conv+BN+ReLU, GEMM+bias+activation become one kernel	Fewer launches, less VRAM traffic (Vol. 06 bandwidth)
Tactic selection	Benchmark candidate CUDA kernels per layer/shape	Build time cost; engine is GPU-SKU specific
Precision	FP32→FP16/BF16/FP8/INT8 with optional calibration	FP16, INT8, this module’s Quantization
Memory planning	Reuse activation buffers; bound workspace	Same pool as VRAM
LLM extras (TRT-LLM)	Paged / inflight KV, speculative decode hooks	KV cache, continuous batching

Eager PyTorch vs TensorRT vs TensorRT-LLM

PyTorch eager / compile

Fast to ship; dynamic Python graphs
Great for research + LoRA iterate
Leaves kernel fusion on the table

TensorRT (vision/NLP encoder)

Classic path: ONNX export → engine
Fixed or limited dynamic axes
Huge wins on CNN/ViT/BERT-class graphs

TensorRT-LLM

Decoder LLMs, KV, inflight batching
Competes with vLLM on NVIDIA fleets
Ops-heavy: rebuild per SKU/version

Minimal Build Sketch (ONNX → Engine)

Production code uses TensorRT Python API or trtexec. The pattern is always: parse → config precision/shapes → build → serialize. Do not check the .plan into git as if it were architecture-agnostic. Pair with Module 18.2 Docker so the builder image matches the runtime GPU driver.

# Sketch — TensorRT Python API (install tensorrt matching CUDA/driver)
import tensorrt as trt

LOGGER = trt.Logger(trt.Logger.WARNING)
builder = trt.Builder(LOGGER)
network = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH))
parser = trt.OnnxParser(network, LOGGER)

with open("encoder.onnx", "rb") as f:
    if not parser.parse(f.read()):
        for i in range(parser.num_errors):
            print(parser.get_error(i))
        raise SystemExit("ONNX parse failed")

config = builder.create_builder_config()
config.set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, 2 << 30)  # 2 GiB
config.set_flag(trt.BuilderFlag.FP16)  # Vol. 06 FP16 tensor cores

engine = builder.build_serialized_network(network, config)
open("encoder_fp16.plan", "wb").write(engine)
# Rebuild on H100 vs 4090; do not copy .plan across CC / TRT versions.
# LLM serving: prefer TensorRT-LLM build pipeline, not this tiny encoder path.

When TensorRT Pays Rent

Reach for TensorRT when

Stable graph, NVIDIA-only fleet, latency SLO is tight
Vision / embedding / reranker encoders at high QPS
You already export clean ONNX (next lecture)
Hopper FP8 or INT8 calibration is validated on a holdout

Skip (for now) when

Weights change daily (LoRA A/B, research)
Dynamic control flow ONNX cannot express cleanly
Multi-vendor or edge NPUs → ONNX Runtime / CoreML
Tier 1 API-only: no local engine to compile

Related Lectures

Lecture	Why it sits beside TensorRT
CUDA	Runtime TensorRT kernels execute on
ONNX	Usual interchange format into the parser
Vol. 06 FP16 / INT8	Precision flags and calibration theory
Vol. 12 KV Cache	What TensorRT-LLM must still budget
FlashAttention	Same I/O idea; different compiler
18.2 Docker	Pin TRT + driver + engine builder images

Common Misconception

“A TensorRT engine is a portable model file.” It is a compiled plan for one GPU family + TRT version + precision. Copying a 4090 .plan onto an A100 is undefined. Second: “INT8 TensorRT is free accuracy.” Calibration (or QAT) can drop metrics; measure like Vol. 19 will demand. Third: “TensorRT replaces FlashAttention / vLLM.” For LLMs you choose an engine family (TRT-LLM vs vLLM vs llama.cpp); they all still implement Vol. 12 KV + batching ideas.

Knowledge Check

Short Answer: What artifact does TensorRT build, and is it a checkpoint? Answer: A serialized engine/plan; no—it is a compiled inference graph, not trained weights alone.
True/False: A TensorRT engine built on RTX 4090 is guaranteed to load on A100. Answer: False—engines are GPU/TRT-version specific.
Multiple Choice: TensorRT usually ingests: (a) CSS, (b) ONNX (or framework parsers), (c) Redis dumps. Answer: (b).
Short Answer: Name two compiler optimizations TensorRT applies. Answer: Layer fusion, tactic/kernel auto-tune, precision conversion, memory reuse (any two).
True/False: TensorRT-LLM still must budget a KV cache for decode. Answer: True (Vol. 12 still applies).
Multiple Choice: Best first TensorRT target: (a) wildly dynamic research LoRA, (b) stable vision/embedding encoder at high QPS, (c) a laptop with no NVIDIA GPU. Answer: (b).
Short Answer: Why is build time long compared to from_pretrained? Answer: Tactic search / kernel benchmarking per layer and shape.
True/False: FP16 TensorRT flags use the same tensor-core idea as Vol. 06 FP16. Answer: True.
Multiple Choice: INT8 TensorRT without calibration/QAT: (a) always lossless, (b) can hurt accuracy—must validate, (c) deletes VRAM. Answer: (b).
Short Answer: When should a team stay on vLLM or API instead of TRT-LLM? Answer: Rapid weight changes, multi-vendor GPUs, or no NVIDIA ops budget / Tier 1 API-only.

Key Takeaways

TensorRT compiles graphs into NVIDIA-specific engines; rebuild per SKU/version/precision.
Wins come from fusion, tactics, dtype, and memory planning—Vol. 06 + 12 ideas as compiler passes.
ONNX is the usual door in; TensorRT-LLM is the LLM-specialized door.
Not a substitute for measuring accuracy after INT8/FP8.
Continue with ONNX—the portable IR behind most TRT builds.

Trainer’s Guide

Lab: Export a tiny Linear/CNN to ONNX (next lecture’s script works). If a CUDA box is available, run trtexec --onnx=... --fp16 --saveEngine=... and compare latency vs PyTorch eager on a fixed batch. If no GPU, walk the parse/build/serialize diagram on paper.

Discussion: For a RAG reranker vs a 70B chat model, which belongs in TensorRT first, and why does KV/inflight batching change the answer?

Recap: TensorRT is NVIDIA’s inference compiler; engines are SKU-specific. Continue with ONNX.