Streaming Token Usage — Module 13.3

How This Lesson Fits the Module & Volume

Streaming improves UX but complicates usage accounting: tokens arrive in deltas, and some SDKs report usage only on the final chunk. Cost dashboards must aggregate correctly.

This lesson connects decode-time output growth to live meters before Module 13.4 billing topics.

Related foundations: recount tokens with tiktoken (Vol. 12) and keep payloads inside the context window (Vol. 11).

Learning Objectives

By the end of this lesson, students should be able to:

Explain how streamed completions emit token deltas.
Aggregate output token counts across chunks.
Locate usage on final streaming events when provided.
Estimate partial cost mid-stream for spend guards.
Avoid double-counting when both delta and final usage exist.
Optionally pre-count prompts with tiktoken before streaming starts.

Definition

Streaming token usage is measuring input and output tokens when completions are delivered incrementally (SSE/WebSockets), including how and when usage metadata appears.

Moment	What you know	Action
Before request	Prompt tokens (local/{TIK})	Authorize against budget
During stream	Growing output estimate	Optional early abort
Final event	Provider usage object	Bill + reconcile

Code: Aggregate Stream + Final Usage

# Illustrative pattern — adapt to your SDK
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")

prompt = "Explain TPM vs context limits briefly."
prompt_tokens = len(enc.encode(prompt))
out_text = []
usage = None

for event in stream:  # pseudo
    if event.text:
        out_text.append(event.text)
    if event.usage:          # often only on last chunk
        usage = event.usage

completion = "".join(out_text)
approx_out = len(enc.encode(completion))
final_out = usage.completion_tokens if usage else approx_out
print(prompt_tokens, final_out, "approx", approx_out)

Non-stream

Usage in one response
Simpler accounting
Higher time-to-first-token

Stream

Better UX
Usage timing varies
Need aggregation logic

Hybrid meter

Local approx + final usage
Best of both
Reconcile diffs

Strengths

Users see tokens early
Enables mid-stream cancel
Still reconciles to provider

Tradeoffs

SDK differences bite
Approx mid-stream can drift
Easy to double-count

Common Misconception

“Streaming means tokens are free until the end.” Output tokens still accrue as they generate; abort policies and max_tokens still apply.

Knowledge Check

Short Answer: When do many APIs attach usage in streams? Answer: Often on the final chunk/event.
True/False: You can estimate output tokens mid-stream with a local encoder. Answer: True.
Multiple Choice: Double-counting risk comes from: (a) summing deltas and final usage naively, (b) using CSS, (c) lowering temperature. Answer: (a).
Short Answer: What should you count before streaming starts? Answer: Prompt/input tokens.
True/False: Streaming disables max_tokens. Answer: False.
Multiple Choice: Mid-stream spend guards need: (a) growing output estimates, (b) only CPU fan speed, (c) DPI. Answer: (a).
Short Answer: Why reconcile approx vs final usage? Answer: Framing/special tokens and encoder skew.
Short Answer: Which Vol. 12 library helps local approx? Answer: tiktoken.
Multiple Choice: Aborting a stream: (a) can stop further output tokens, (b) refunds the whole internet, (c) deletes the model. Answer: (a).
True/False: Usage accounting is optional for streamed endpoints. Answer: False.

Key Takeaways

Stream UX ≠ free tokens.
Pre-count prompts; aggregate outputs.
Prefer final provider usage for billing.
Support mid-stream abort on budget.
Next: Token Overflow Handling.

Trainer’s Guide

Lab: Log a streamed response; implement approx_out vs final usage diff alert if >2%.

Discussion: Should clients show live token counters to end users?

Recap: Streaming needs deliberate usage aggregation. Continue with Token Overflow Handling.