← Master Index
Vol. 18 Module 18.4 Lecture

Tier 2 — Local Basic (16GB RAM, RTX 3060/4060, quantized 7B models INT4/INT8)

System Requirements — Basic to Advanced (added)

How This Lesson Fits the Module & Volume

Tier 1 kept weights in the cloud. Tier 2 is the first honest local GPU rung: ~16 GB system RAM, an RTX 3060/4060-class card (8–16 GB VRAM), and quantized 7B models in INT8 / INT4. Module 18.3 quantization is not optional here—FP16 7B (~14 GB weights) plus KV cache will OOM a 12 GB card. FlashAttention (if your engine ships it) buys batch/context headroom, not a 13B FP16 miracle.

This is hobby, privacy sandbox, and small internal tools—not a 34B FP16 workstation (Tier 3) and not QLoRA-as-a-service (Tier 4).

Learning Objectives

By the end of this lesson, students should be able to:

  • Write a Tier 2 BOM: 16 GB RAM, 8–16 GB VRAM GPU, NVMe, CUDA driver + wheel.
  • Budget 7B INT4 vs INT8 vs FP16 including KV at 2k/8k context.
  • Choose an engine: llama.cpp/Ollama/GGUF vs HF+bitsandbytes vs small vLLM.
  • Know what does not fit: 13B+ FP16, serious LoRA train, multi-user 32k context.
  • Apply Vol. 06 VRAM + Vol. 12 KV before blaming the GPU shop listing.
  • Decide stay-API vs Tier 2 with a quality + privacy + QPS argument.
Definition

Tier 2 (Local Basic) is a single consumer NVIDIA GPU (RTX 3060/4060 class, typically 8–12 GB, sometimes 16 GB 4060 Ti) plus ~16 GB host RAM, used to infer ~7B dense LLMs in weight-only INT4 or INT8, short-to-medium context, single or few concurrent users. Training is limited to toy LoRA or not at all. System RAM holds OS, tokenizer, offload spill; VRAM holds quantized weights + KV + CUDA tax.

VRAM Budget (7B, single user)

Weights from Module 18.3: ~14 GB FP16, ~7 GB INT8, ~4 GB INT4. Add CUDA context (~1 GB) and KV. Rough KV (GQA 7B-like): 32 layers × seq × 8 kv_heads × 128 dim × 2 (K+V) × 2 bytes FP16 ≈ 1 GB at 4k seq, ~2 GB at 8k. A 12 GB card: INT4 7B + 4k ctx is comfortable; FP16 7B + 4k is a tight or failed fit.

SetupWeightsKV ~4k12 GB card?8 GB card?
7B FP16~14 GB~1 GBNoNo
7B INT8~7 GB~1 GBMaybe (tight)Unlikely
7B INT4 / GGUF Q4~4 GB~1 GBYesYes if context modest
13B INT4~7–8 GB~1.5+ GBBorderlineUsually no
7B INT4, 32k ctx, 4 users~4 GBKV explodesOften OOMNo—Tier 3 or API

Software Stack on This Box

llama.cpp / Ollama / GGUF

  • Best everyday Tier 2 DX
  • CPU offload if VRAM short
  • Not the QLoRA train path

HF + bitsandbytes

  • Matches 18.3 quant lesson
  • Slower decode than GGUF/vLLM
  • Stepping stone to QLoRA (Tier 4)

vLLM GPTQ/AWQ 7B

  • If you want OpenAI-compatible serve
  • Needs FA + paged KV (Vol. 12)
  • Overkill for one-user desktop chat

Inventory + Tiny 4-bit Load

Confirm CUDA first (18.3). Then load INT4—do not start from FP16 and “see what happens.”

import torch from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig assert torch.cuda.is_available(), "Tier 2 expects an NVIDIA GPU + CUDA wheel" p = torch.cuda.get_device_properties(0) print(p.name, "VRAM GiB", round(p.total_memory / 1024**3, 2)) # Expect ~8–12 (3060/4060) or 16 (4060 Ti). 24 GB is already Tier 3 silicon. bnb = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.float16, # 3060: FP16; 30-series BF16 ok on some bnb_4bit_use_double_quant=True, ) name = "mistralai/Mistral-7B-Instruct-v0.2" tok = AutoTokenizer.from_pretrained(name) model = AutoModelForCausalLM.from_pretrained(name, quantization_config=bnb, device_map="auto") print("alloc GiB", round(torch.cuda.memory_allocated() / 1024**3, 2)) # Everyday DX alternative (shell): ollama run mistral:7b-instruct-q4_K_M # Keep context 2k–8k. 32k + multi-user → KV OOM; go Tier 3 or stay API (Tier 1).

Stay API vs Buy This Card

Tier 2 is worth it when

  • Privacy sandbox / offline demo for 7B-quality tasks
  • Token spend on a small model already exceeds a 3060 TCO
  • You need to learn CUDA/quant without a workstation
  • Single user, ≤8k context, INT4 quality is enough

Stay Tier 1 or jump to 3+

  • You still need frontier quality → API
  • 13B+ FP16 or long-context multi-user → Tier 3
  • Real LoRA/QLoRA programs → Tier 4
  • No ops appetite for drivers → stay API

Related Lectures

LectureWhy it sits beside Tier 2
18.3 GPU / CUDASilicon + driver stack
18.3 QuantizationWhy INT4 exists on this tier
Vol. 06 VRAM / INT8Bytes/param theory
Vol. 12 KV CacheContext is the silent OOM
FlashAttentionHelps peak mem if the engine has it
Tier 3FP16 13B–34B headroom
Common Misconception

“A 3060 runs any 7B at FP16.” Weight math says ~14 GB; the card has 12 GB or less. INT4/INT8 is the tier, not a bonus. Second: “16 GB RAM means 16 GB VRAM.” System RAM ≠ GPU memory; offload is slow. Third: “Tier 2 is for fine-tuning Llama-70B.” At most tiny LoRA experiments; serious FT is Tier 4.

Knowledge Check

  1. Short Answer: What model class and dtypes define Tier 2? Answer: ~7B LLMs in INT4/INT8 (quantized), consumer 8–16 GB GPU, ~16 GB host RAM.
  2. True/False: FP16 Mistral-7B comfortably fits a 12 GB 3060 with 4k context. Answer: False—weights alone ~14 GB.
  3. Multiple Choice: Best everyday Tier 2 chat DX: (a) H100 NCCL, (b) Ollama/llama.cpp Q4, (c) DeepSpeed ZeRO-3 70B. Answer: (b).
  4. Short Answer: Why can INT4 7B still OOM at 32k context with several users? Answer: KV cache scales with seq × batch; weights were not the only resident.
  5. True/False: Host 16 GB RAM can replace 12 GB VRAM for fast decode. Answer: False—CPU offload is much slower; VRAM is the pool that matters.
  6. Multiple Choice: 13B FP16 belongs on: (a) Tier 2 3060, (b) Tier 3 24–48 GB, (c) an MCU. Answer: (b).
  7. Short Answer: Name one Vol. 12 feature that helps Tier 2 engines. Answer: Paged KV, FlashAttention, or continuous batching (any).
  8. True/False: bitsandbytes NF4 load is valid on Tier 2 but often slower than GGUF for chat. Answer: True.
  9. Multiple Choice: If you need frontier quality and have no privacy constraint: (a) force INT4 7B, (b) stay Tier 1 API, (c) buy InfiniBand. Answer: (b).
  10. Short Answer: What RAM figure in the tier title is host RAM, not VRAM? Answer: 16 GB system RAM; the GPU is typically 8–16 GB VRAM separately.

Key Takeaways

  • Tier 2 = consumer 8–16 GB GPU + 16 GB RAM + quantized 7B (INT4/INT8).
  • Budget weights + KV + CUDA tax; FP16 7B is not this tier.
  • Ollama/GGUF for chat; bnb for learning quant/QLoRA later.
  • Long context and multi-user push you to Tier 3 or back to the API.
  • Continue with Tier 3 Local Advanced.
Trainer’s Guide

Lab: Spreadsheet a 3060 12 GB budget: 7B Q4 vs INT8 vs FP16 at 2k/8k/32k, 1 vs 4 users. If a GPU is available, load Q4 via Ollama and watch nvidia-smi during a long prompt.

Discussion: A founder wants “ChatGPT on every employee 3060.” Separate quality, context, concurrency, and compliance—then pick Tier 1, 2, or 3 per workload.

Recap: Tier 2 runs quantized 7B models on consumer GPUs. Continue with Tier 3 Local Advanced.