← Master Index
Vol. 18 Module 18.4 Lecture

Tier 3 — Local Advanced (24–48GB VRAM, RTX 4090 / A6000, 13B–34B models, FP16/BF16)

System Requirements — Basic to Advanced (added)

How This Lesson Fits the Module & Volume

Tier 2 survived on INT4 7B. Tier 3 is the workstation infer box: 24–48 GB VRAM (RTX 4090 24 GB, RTX 6000 Ada / A6000 48 GB), running 13B–34B in FP16 / BF16—or larger models still quantized. Volume 12 KV cache and FlashAttention decide whether that 24 GB feels like 18 GB usable. This is still mostly inference + light PEFT experiments, not multi-GPU QLoRA farms (Tier 4) or NCCL clusters (Tier 5).

Learning Objectives

By the end of this lesson, students should be able to:

  • Map 24 GB vs 48 GB to 13B/34B FP16 fit, including KV headroom.
  • Explain why BF16 is preferred on Ampere+ workstation/datacenter vs FP16 on some consumer paths.
  • Size vLLM / TensorRT-LLM / HF serve for a handful of concurrent users.
  • Know when 70B INT4 on 48 GB is in-tier vs when 70B FP16 is not.
  • Contrast 4090 (fast consumer) vs A6000 (capacity, ECC, workstation).
  • Draw the line to Tier 4 when LoRA/QLoRA becomes the job, not chat.
Definition

Tier 3 (Local Advanced) is a single high-end GPU with 24–48 GB of VRAM used primarily for high-quality local inference of mid-size LLMs in FP16/BF16 (13B class comfortably; 34B class on 48 GB or with quant on 24 GB) plus embeddings, vision towers, and modest speculative-decode drafts. Host RAM should be 64 GB-class so dataloaders and tokenizers are not fighting the GPU. Multi-GPU NVLink training is out of scope unless you are peeking at Tier 4.

Fit Table (weights FP16/BF16 ≈ 2 bytes/param)

ModelFP16 weights4090 24 GBA6000 48 GBNotes
7B FP16~14 GBEasy + long KV / batchEasy; overkillTier 2 could already INT4 this
13B FP16~26 GBNo FP16; yes INT8/INT4 or 2×GPUYes + KV room4090 often runs 13B quantized
34B FP16~68 GBNoNo FP16; INT4/INT8 maybe34B FP16 wants 80 GB or multi-GPU
34B INT4~18–22 GBTight yes (watch KV)ComfortableQuality vs 13B FP16 is a product call
70B INT4~35–40 GBNoMaybe + short ctx70B FP16 is Tier 5 / 2×80 GB

Read the table slowly: the module title’s “13B–34B FP16/BF16” is the ambition band, not a promise that every 34B FP16 fits a 4090. 48 GB is the card that makes 13B FP16 honest; 34B FP16 usually still needs quant or more silicon. That honesty is the playbook.

4090 vs A6000 vs “just quantize”

RTX 4090 (24 GB)

  • Huge Ada tensor-core throughput
  • Best $/infer for 7B FP16, 13B INT4, SDXL
  • No ECC; 24 GB ceiling; noisy desktop

A6000 / 6000 Ada (48 GB)

  • 13B FP16 + real KV/batch
  • ECC, quieter, multi-GPU PCIe possible
  • Slower than 4090 on some infer kernels

Stay quantized anyway

  • 34B/70B on one card almost always INT4/INT8
  • FP16 is for quality-sensitive 7–13B
  • Vol. 19 will make you measure, not assume

Serve Sketch (vLLM-minded)

Tier 3 is where continuous batching, paged KV, and FlashAttention stop being trivia and start being capacity. Pin dtype explicitly.

import torch p = torch.cuda.get_device_properties(0) vram = p.total_memory / 1024**3 print(p.name, f"{vram:.1f} GiB", "CC", f"{p.major}.{p.minor}") # Dtype policy (Vol. 06): # A100/A6000/Ada: prefer BF16 for infer/train stability # 4090: BF16 or FP16; measure; FA kernels vary by build dtype = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16 # Capacity sketch — do not treat as a guarantee: # usable ≈ 0.80 * vram (CUDA tax + fragmentation) # weights_gb ≈ params_b * (2 if fp16/bf16 else 0.5 if int4) # kv_gb ≈ 2 * n_layers * seq * kv_heads * head_dim * bytes * batch / 1e9 # if weights_gb + kv_gb + 2 > 0.80 * vram: quantize, shorten ctx, or go multi-GPU / API print("bf16 supported", torch.cuda.is_bf16_supported(), "working dtype", dtype) # vLLM (shell): vllm serve meta-llama/Llama-3.1-8B-Instruct --dtype bfloat16 --max-model-len 8192 # 13B FP16 on 24 GB: usually --quantization awq|gptq or a 48 GB card. # TensorRT-LLM: same KV math, SKU-specific engine (Module 18.3).

When Tier 3 Is the Wrong Spend

Buy/rent Tier 3 when

  • Local 13B FP16 quality or 34B INT4 privacy sandbox
  • SDXL / FLUX FP16 (Vol. 17) plus a small LLM on 24–48 GB
  • A few internal users, OpenAI-compatible vLLM on-LAN
  • You outgrew 3060 KV limits, not just “want RGB”

Do not

  • Expect 70B FP16 on one 4090
  • Run serious multi-epoch QLoRA on 34B as if this were Tier 4
  • Ignore PSU/thermals (4090 450 W class)
  • Skip Tier 1 if API quality still dominates TCO

Related Lectures

LectureWhy it sits beside Tier 3
Vol. 06 FP16 / BF16The dtypes in the tier name
Vol. 06 VRAM24 vs 48 GB is the whole game
KV Cache / FlashAttentionUsable capacity vs brochure GB
TensorRTOptional compile path on this SKU
Vol. 12 LoRALight adapters possible; heavy FT → Tier 4
Tier 4When training is the job
Common Misconception

“24 GB VRAM runs any 13B FP16.” 13B × 2 bytes ≈ 26 GB weights before KV. A 4090 runs 13B well in quant or with offload; 13B FP16 is an A6000-class story. Second: “34B FP16 on 48 GB.” 34B × 2 ≈ 68 GB. Third: “BF16 and FP16 are interchangeable everywhere.” Vol. 06: BF16 has FP32-range exponents; consumer FA/TensorRT builds differ—measure.

Knowledge Check

  1. Short Answer: What VRAM band defines Tier 3? Answer: About 24–48 GB (4090 / A6000 class).
  2. True/False: Llama-13B FP16 weights alone fit in 24 GB with room for 8k KV. Answer: False—weights ~26 GB already exceed 24 GB.
  3. Multiple Choice: Honest 13B FP16 single-GPU home: (a) 3060 12 GB, (b) A6000 48 GB, (c) phone NPU. Answer: (b).
  4. Short Answer: ~FP16 size of a 34B model? Answer: ~68 GB (34e9 × 2 bytes).
  5. True/False: 34B INT4 can be in-tier on a 24–48 GB card with short/medium context. Answer: True (tight on 24 GB).
  6. Multiple Choice: Why FlashAttention matters more here than on Tier 1: (a) CSS, (b) usable activation/KV headroom on a finite GPU, (c) API keys. Answer: (b).
  7. Short Answer: Name one 4090 vs A6000 trade. Answer: 4090 faster/24 GB vs A6000 48 GB + ECC/workstation (or thermals/PSU).
  8. True/False: 70B FP16 is a Tier 3 single-card workload. Answer: False—needs ~140 GB weights or multi-80 GB / quant+pipeline (Tier 5-ish).
  9. Multiple Choice: Preferred dtype on A100/A6000 infer: (a) FP64, (b) BF16 when supported, (c) INT32. Answer: (b).
  10. Short Answer: When do you leave Tier 3 for Tier 4? Answer: When LoRA/QLoRA (or heavier FT) is the primary job, especially multi-GPU 48–80 GB.

Key Takeaways

  • Tier 3 = 24–48 GB infer workstation; 13B FP16 wants 48 GB; 34B FP16 is not a 4090 promise.
  • Quant still matters; FP16/BF16 is for the models that actually fit.
  • vLLM + FA + paged KV turn brochure GB into concurrent users.
  • 4090 = speed; A6000 = capacity/ECC.
  • Continue with Tier 4 Fine-Tuning Workstation.
Trainer’s Guide

Lab: Students compute usable VRAM at 80% for 24 vs 48 GB, then place 8B/13B/34B/70B at FP16 vs INT4 with 4k and 16k KV. Debate one purchase: 4090 vs used A6000 vs stay API.

Discussion: Is “34B FP16 on Tier 3” marketing or math? Make them say quant, multi-GPU, or smaller model.

Recap: Tier 3 is 24–48 GB local infer for mid-size FP16/BF16 (and quantized larger) models. Continue with Tier 4.