← Master Index
Vol. 18 Module 18.4 Lecture

Tier 4 — Fine-Tuning Workstation (multi-GPU, 48–80GB VRAM per card, LoRA/QLoRA training)

System Requirements — Basic to Advanced (added)

How This Lesson Fits the Module & Volume

Tier 3 was local inference. Tier 4 is when the job is training adapters: LoRA and QLoRA on a workstation with 48–80 GB per card, often 2–4 GPUs (A6000, L40S, A100 80 GB, RTX 6000 Ada). Volume 06 optimizer states and mixed precision come back; Volume 12 PEFT is the method. Full 70B FT still belongs on Tier 5.

If you only chat with a 13B, you overbought. If you QLoRA 70B on a 3060, you underbought. This lecture is that sizing line.

Learning Objectives

By the end of this lesson, students should be able to:

  • Contrast infer VRAM vs train VRAM (activations + Adam + grads vs weights + KV).
  • Size QLoRA 7B/13B/70B vs LoRA-FP16 vs full FT on 48 vs 80 GB.
  • Explain why multi-GPU (DDP) helps batch/seq more than “more CUDA cores.”
  • Sketch a PEFT + bitsandbytes training loop that matches Vol. 12 QLoRA.
  • List workstation extras: 128+ GB host RAM, NVMe, cooling, PCIe topology.
  • Know when to rent a cloud 80 GB box instead of buying four A6000s.
Definition

Tier 4 (Fine-Tuning Workstation) is a 1–4 GPU machine with 48–80 GB VRAM per card, provisioned to train PEFT adapters (LoRA/QLoRA) and occasionally small full fine-tunes. Inference still works (it is a superset of Tier 3), but the BOM is driven by optimizer + activation memory, not by decode KV alone. Interconnect is typically PCIe (sometimes NVLink on dual A6000); it is not an InfiniBand pod.

Train Memory vs Infer Memory

ResidentInference (Tiers 2–3)LoRA FP16QLoRA (NF4 base)Full FT Adam
Base weightsFP16/INT4FP16 frozenNF4 frozen (~0.5 B/param)FP16/BF16 trainable
Adapters—Small FP16 BA + grads + AdamSame on adaptersN/A (all W)
ActivationsPrefill scratch; FA helpsSeq × batch; checkpointing optionalSameSame, often larger batch
KV cacheDominant at decodeNot the train bottleneckNot the train bottleneckNot the train bottleneck
Adam statesNoneOnly on LoRA paramsOnly on LoRA params~8–16 B/param extra

Rule-of-Thumb Fit (single GPU, QLoRA SFT)

48 GB (A6000 class)

  • QLoRA 7B/13B comfortable (2k–8k)
  • QLoRA 34B possible with short seq / checkpoint
  • LoRA-FP16 13B tight; 34B usually no
  • Full FT 7B maybe; 13B+ no

80 GB (A100/H100 80)

  • QLoRA 70B is the headline use
  • LoRA-FP16 13B–34B realistic
  • Full FT 13B still painful without ZeRO
  • Rent before buying a pile of 80s

2–4× GPU DDP

  • Larger global batch / faster epochs
  • Does not magically shard a 70B full FT
  • PCIe is enough for LoRA; NVLink nicer
  • FSDP/ZeRO → you are peeking at Tier 5

QLoRA Sketch on a Workstation

Same bitsandbytes config as Module 18.3; now we attach PEFT and train. Gradient checkpointing trades compute for activation VRAM—mandatory on long seq.

import torch from transformers import AutoModelForCausalLM, BitsAndBytesConfig, TrainingArguments, Trainer from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training bnb = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True, ) model = AutoModelForCausalLM.from_pretrained( "mistralai/Mistral-7B-Instruct-v0.2", quantization_config=bnb, device_map="auto", ) model = prepare_model_for_kbit_training(model) model = get_peft_model(model, LoraConfig( r=16, lora_alpha=32, lora_dropout=0.05, target_modules=["q_proj", "k_proj", "v_proj", "o_proj"], task_type="CAUSAL_LM", )) model.gradient_checkpointing_enable() # TrainingArguments(..., bf16=True, per_device_train_batch_size=1, # gradient_accumulation_steps=16, learning_rate=2e-4, ddp_find_unused_parameters=False) # Multi-GPU: torchrun --nproc_per_node=2|4 train.py # 70B QLoRA: start on 80 GB; do not expect a 4090 24 GB to be "almost" this tier.

Buy, Rent, or Stay API + Hosted FT

Own/rent Tier 4 when

  • Private corpora cannot hit a public FT API
  • You iterate LoRA weekly (productization)
  • Several mid-size jobs share the box (queue)
  • You already have 18.2 Docker + experiment tracking

Skip owning silicon

  • One-off SFT: rent A100 80 GB for a weekend
  • Hosted FT APIs (Tier 1-adjacent) meet the DPA
  • You only needed better prompts, not weights
  • Full 70B FT / multi-node → Tier 5, not a desk

Related Lectures

LectureWhy it sits beside Tier 4
Vol. 12 LoRA / QLoRAThe methods you are sizing for
Full fine-tuningWhy Adam blows past 48 GB
Vol. 06 Mixed PrecisionBF16 train on Ampere+
18.3 QuantizationNF4 base load
Vol. 06 Distributed TrainingDDP vs real sharding
Tier 5NVLink/IB clusters
Common Misconception

“If 7B QLoRA fits 16 GB, 70B QLoRA fits 16×10 = wait no, it scales with params.” 70B NF4 weights alone are ~35–40 GB; plus activations you want 80 GB. Second: “Two 24 GB 4090s equal one 48 GB for QLoRA 34B.” Without tensor/pipeline parallel you have two separate 24 GB pools—DDP replicates the model. Third: “LoRA train VRAM ≈ infer VRAM.” Activations + adapter Adam dominate; checkpointing is not optional folklore.

Knowledge Check

  1. Short Answer: What VRAM-per-card band defines Tier 4? Answer: About 48–80 GB per GPU, often multi-GPU.
  2. True/False: Tier 4 is primarily an inference tier like Tier 3. Answer: False—it is sized for LoRA/QLoRA training (infer is a bonus).
  3. Multiple Choice: QLoRA 70B single-GPU home: (a) 3060 12 GB, (b) A100 80 GB, (c) Raspberry Pi. Answer: (b).
  4. Short Answer: Name two extra VRAM residents in training vs infer. Answer: Activations (backward), gradients, optimizer states (any two).
  5. True/False: DDP on 2×24 GB automatically shards a 34B FP16 base across cards. Answer: False—DDP replicates; you need TP/PP/ZeRO to shard.
  6. Multiple Choice: QLoRA base dtype: (a) NF4/4-bit + LoRA adapters, (b) FP64 full W, (c) CSS. Answer: (a).
  7. Short Answer: Why enable gradient checkpointing on long sequences? Answer: Recompute activations to cut peak VRAM.
  8. True/False: Full FT Adam on 13B typically fits a single 48 GB card comfortably. Answer: False—optimizer+grads usually demand ZeRO/FSDP or 80 GB+ / smaller model.
  9. Multiple Choice: One-off weekend SFT with no data-residency issue: (a) always buy 4×A6000, (b) rent 80 GB cloud, (c) train on a phone NPU. Answer: (b).
  10. Short Answer: When do you leave Tier 4 for Tier 5? Answer: Multi-node distributed training, NVLink/IB fabrics, full large-model FT / pretrain.

Key Takeaways

  • Tier 4 = 48–80 GB/card workstation for LoRA/QLoRA, not just bigger chat.
  • Train memory ≠ infer memory; checkpointing and PEFT are the levers.
  • DDP replicates; sharding is Tier 5 territory.
  • Rent 80 GB for one-offs; own silicon for weekly private FT.
  • Continue with Tier 5 Production Cluster.
Trainer’s Guide

Lab: Size three jobs on paper: (1) QLoRA 7B seq 2048, (2) QLoRA 70B seq 4096, (3) full FT 13B Adam. Assign each to 12 / 24 / 48 / 80 GB or “cluster.” If GPUs exist, run a 10-step QLoRA 7B and record nvidia-smi vs infer-only load.

Discussion: Should a startup buy dual A6000s or use a hosted FT API + Tier 1 serving? Include data residency and iteration cadence.

Recap: Tier 4 sizes LoRA/QLoRA workstations at 48–80 GB per card. Continue with Tier 5.