Rate Limiting — Module 13.4

How This Lesson Fits the Module & Volume

Providers enforce rate limits (RPM, TPM, concurrency) to protect capacity. Your app should also rate-limit tenants to protect the budget and fairness.

Links 13.3 token overflow (429 TPM) to cost control: throughput caps are spend caps in disguise.

Related foundations: recount tokens with tiktoken (Vol. 12) and keep payloads inside the context window (Vol. 11).

Learning Objectives

By the end of this lesson, students should be able to:

Define RPM, TPM, and concurrency limits.
Implement client backoff with jitter on 429s.
Throttle per-tenant to protect shared quotas.
Separate provider limits from product fair-use limits.
Monitor retry amplification as a cost risk.
Design queues for bursty workloads.

Definition

Rate limiting restricts how many requests or tokens may be processed per unit time—by the provider, API gateway, or your application—to ensure stability and fair spend.

Limit	Meters	Typical signal
RPM	Requests / minute	429 + retry-after
TPM	Tokens / minute	429; reduce payload/rate
Concurrency	In-flight calls	Queue or shed load
App fair-use	Per user/org	429/402 from your API

Code: Exponential Backoff Sketch

import random, time

def call_with_backoff(send, max_tries=6):
    delay = 0.5
    for attempt in range(max_tries):
        status, body = send()
        if status != 429:
            return body
        sleep = delay * (1 + random.random())
        time.sleep(sleep)
        delay = min(delay * 2, 30)
    raise RuntimeError("rate limited too long")

# Pair with token-aware client throttles:
# tokens_in_flight + new_prompt_tokens <= TPM_budget

Provider limit

Shared capacity
Not optional
Upgrade tier / shape traffic

App limit

Protect COGS/UX
Policy choice
Per-tenant keys

Backoff

Survives spikes
Adds latency
Needs jitter

Strengths

Prevents cascading outages
Caps surprise TPM spend
Enables multi-tenant fairness

Tradeoffs

Naive retries amplify load
UX waits on queues
Mis-tuned limits false-throttle

Common Misconception

“On 429, immediately retry as fast as possible in a tight loop.” That stampedes the API. Use exponential backoff, jitter, and client-side token buckets.

Knowledge Check

Short Answer: What is TPM? Answer: Tokens per minute (provider or app throughput cap).
True/False: RPM counts tokens. Answer: False—requests per minute.
Multiple Choice: Healthy 429 handling uses: (a) tight spin loops, (b) backoff + jitter, (c) delete the key. Answer: (b).
Short Answer: Why per-tenant limits? Answer: Stop one customer from burning shared quota/budget.
True/False: Rate limits can indirectly cap spend. Answer: True.
Multiple Choice: Retry amplification risks: (a) lower load, (b) higher load/cost, (c) free GPUs. Answer: (b).
Short Answer: Name a client-side tool for pacing. Answer: Token bucket / queue / semaphore.
Short Answer: How does this relate to 13.3 overflow? Answer: 429 TPM is a throughput overflow signal.
Multiple Choice: Concurrency limits cap: (a) in-flight calls, (b) vocabulary size, (c) CSS. Answer: (a).
True/False: App fair-use limits must equal provider RPM. Answer: False—often stricter.

Key Takeaways

Respect RPM/TPM/concurrency.
Backoff with jitter; throttle locally.
Fair-use protects multi-tenant COGS.
Watch retries as cost amplifiers.
Next: Usage Quotas.

Trainer’s Guide

Lab: Simulate 429s; compare spin-retry vs backoff success and total attempts.

Discussion: Should power users buy higher product tiers or raw provider keys?

Recap: Rate limits pace both reliability and spend. Continue with Usage Quotas.