← Master Index
Vol. 13 Module 13.4 Lecture

Batch API Discounts

Price & Cost Control (added)

How This Lesson Fits the Module & Volume

Batch APIs trade latency for price: submit jobs asynchronously, get discounted token rates hours later. Ideal for offline evals, backfills, and nightly reports—not live chat.

Another price-tier lever alongside caching and model routing.

Related foundations: recount tokens with tiktoken (Vol. 12) and keep payloads inside the context window (Vol. 11).

Learning Objectives

By the end of this lesson, students should be able to:

  • Identify workloads fit for batch vs real-time.
  • Estimate discount savings vs interactive endpoints.
  • Design idempotent job submission and result pickup.
  • Handle partial failures in large batches.
  • Keep PII/retention policies intact for delayed jobs.
  • Avoid batching user-facing interactive paths.
Definition

Batch API discounts are reduced per-token (or per-job) prices for asynchronous, deferred inference with weaker latency SLOs than synchronous chat APIs.

Good for batchKeep real-timeReason
Eval suitesChat UXHours-ok vs ms-needed
Corpus labelingTool loops in agentsThroughput vs interactivity
Nightly summariesFraud stop-the-lineDelay tolerance
Prompt A/B offlineLive retrieval chatSLO mismatch

Code: Discount Compare

def job_cost(tokens_in, tokens_out, in_rate, out_rate, discount=1.0): return discount * (tokens_in * in_rate + tokens_out * out_rate) / 1e6 realtime = job_cost(2_000, 300, 2.5, 10.0, discount=1.0) batch = job_cost(2_000, 300, 2.5, 10.0, discount=0.5) # example 50% off n = 100_000 print(f"realtime ${realtime*n:,.0f} vs batch ${batch*n:,.0f}")

Realtime API

  • Low latency
  • Full price
  • Interactive

Batch API

  • Deferred
  • Discounted
  • Offline scale

Hybrid

  • Chat live + batch evals
  • Two pipelines
  • Common prod pattern

Strengths

  • Material COGS reduction
  • Smooths provider load
  • Great for eval/backfill

Tradeoffs

  • Not for chat SLOs
  • Operational complexity
  • Delayed error discovery
Common Misconception

“We will batch the customer’s chat replies overnight to save money.” Interactive products need synchronous paths; batch is for work that can wait.

Knowledge Check

  1. Short Answer: What do you trade for batch discounts? Answer: Latency / immediacy (async completion).
  2. True/False: Live chat should default to batch APIs. Answer: False.
  3. Multiple Choice: Strong batch fit: (a) offline evals, (b) typing indicators, (c) WebSocket pings. Answer: (a).
  4. Short Answer: Name an ops concern for batches. Answer: Partial failures, retries, idempotency, retention.
  5. True/False: Discounts still meter tokens. Answer: True.
  6. Multiple Choice: Hybrid architecture: (a) all batch, (b) realtime UX + batch offline, (c) no metering. Answer: (b).
  7. Short Answer: Why idempotent job IDs? Answer: Safe retries without double billing/work.
  8. Short Answer: How does estimation change? Answer: Apply discount factor to the same token formula.
  9. Multiple Choice: Fraud stop-the-line scoring: (a) batch overnight, (b) realtime, (c) ignore tokens. Answer: (b).
  10. True/False: Batch eliminates the need for quotas. Answer: False.

Key Takeaways

  • Batch = cheaper async tokens.
  • Reserve it for delay-tolerant work.
  • Engineer idempotent pickup and failure handling.
  • Compare savings explicitly in estimators.
  • Next: Model Tiering for Cost.
Trainer’s Guide

Lab: Split a workload list into batch vs realtime; estimate 30-day savings at 50% discount.

Discussion: What product copy sets user expectations when reports are batch-generated?

Recap: Batch discounts buy dollars with time. Continue with Model Tiering.