Model Tiering for Cost (small vs large model routing) — Module 13.4

How This Lesson Fits the Module & Volume

Model tiering routes easy requests to small/cheap models and hard ones to large/expensive models. Often the highest-ROI cost control after basic token hygiene.

Combines quality gates with the price matrix—not “always GPT-max.”

Related foundations: recount tokens with tiktoken (Vol. 12) and keep payloads inside the context window (Vol. 11).

Learning Objectives

By the end of this lesson, students should be able to:

Define a tier ladder (small / medium / large).
Design routers: rules, classifiers, or cascaded fallback.
Measure quality vs cost on a shared eval set.
Fail open to larger models on low confidence.
Account for router overhead tokens/calls.
Document which product paths are pinned to which tier.

Definition

Model tiering for cost is routing traffic across models of different capability and price so average COGS falls while meeting quality SLOs via escalation when needed.

Tier	Role	Examples of fit
Small	Classify, extract, rewrite	Intent, short FAQ
Medium	Default assistant	Support with light RAG
Large	Hard reasoning / sensitive	Legal, complex multi-hop
Router	Choose tier	Rules or tiny classifier

Code: Cascade Sketch

PRICE_OUT = {"small": 0.4, "large": 10.0}  # $/1M out illustrative

def route(query: str) -> str:
    # Toy rules — replace with classifier + features
    if len(query) < 80 and query.lower().startswith(("tag:", "label:")):
        return "small"
    if "compare statutes" in query.lower() or "multi-hop" in query.lower():
        return "large"
    return "small"  # try cheap first

def answer(query: str, call):
    tier = route(query)
    draft = call(tier, query)
    if tier == "small" and not confident(draft):
        return call("large", query)  # escalate
    return draft

Always-large

Max quality ceiling
Max COGS
Simple ops

Tiered routing

Lower average $
More moving parts
Needs evals

Cascade

Try cheap then escalate
Extra latency on hard
Strong default

Strengths

Cuts average cost dramatically
Preserves quality via escalation
Matches product criticality

Tradeoffs

Router errors mis-tier
Dual eval burden
Latency on cascades

Common Misconception

“The small model failed once, so everything goes to the large model forever.” Use measured escalation rates; do not abandon tiering after a single miss.

Knowledge Check

Short Answer: What is model tiering? Answer: Routing to cheaper vs costlier models by difficulty/need.
True/False: Escalation on low confidence is a common pattern. Answer: True.
Multiple Choice: Best default for “label:” intents: (a) largest model, (b) small model, (c) no model. Answer: (b).
Short Answer: Why eval both tiers? Answer: Ensure quality SLO while proving cost savings.
True/False: Router calls are always free. Answer: False—they can add tokens/latency.
Multiple Choice: Cascade means: (a) cheap then escalate, (b) only batch, (c) drop quotas. Answer: (a).
Short Answer: Name a risk of tiering. Answer: Mis-routing hard queries to small models.
Short Answer: How do price cards inform tiers? Answer: Larger models usually cost more per token.
Multiple Choice: Pinning legal paths to large models is: (a) reckless always, (b) often correct, (c) about CSS. Answer: (b).
True/False: One failure justifies deleting the small tier. Answer: False.

Key Takeaways

Route easy work cheaply; escalate hard work.
Eval quality and cost together.
Count router overhead.
Pin critical paths explicitly.
Next: Cost Monitoring Dashboards.

Trainer’s Guide

Lab: Propose tiers for a support product; estimate % traffic per tier and blended $/request.

Discussion: Rules vs learned router—when is each safer?

Recap: Tiering is strategic under-spend with an escape hatch. Continue with Cost Monitoring Dashboards.