Completion — Module 11.3

How This Lesson Fits the Module & Volume

A prompt conditions the model; a completion is the generated continuation. This closes Module 11.3 by connecting decoding (Module 11.1 sampling, temperature, top-p) to product APIs, then hands off to Module 11.4’s Large Language Model concepts.

Learning Objectives

By the end of this lesson, students should be able to:

Define completion as sampled tokens after the prompt.
Contrast stop sequences, max tokens, and finish reasons.
Relate greedy, sampling, and beam search to completion quality.
Separate prompt tokens from completion tokens for billing/logging.
Handle streaming completions in UX design.
List failure modes: truncation, repetition, off-format outputs.

Definition

A completion is the sequence of tokens an autoregressive language model generates given a prompt (and decoding parameters). In chat APIs the completion is typically the assistant message content.

Anatomy of a Call

Prompt

Conditioning prefix.

Decode loop

Sample under causal LM.

Stop

EOS, stop string, or max length.

Post-process

Parse, validate, display.

Control	Effect on completion
`max_new_tokens` / max tokens	Hard length cap; may truncate mid-thought
Stop sequences	End when a delimiter appears
Temperature / top-p	Diversity vs. determinism
Presence/frequency penalties	Reduce repetition (API-dependent)

Greedy

Argmax each step
Stable, dull
Good for formats

Sampling

Draw from top-p/k
Creative variety
Needs seeds for repro

Chat completion

Assistant role text
May include tool calls
Multi-turn append

from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("openai-community/gpt2")
model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2")
prompt = "Defensive API design tip #1:"
inputs = tok(prompt, return_tensors="pt")
out = model.generate(
    **inputs,
    max_new_tokens=30,
    do_sample=True,
    top_p=0.9,
    temperature=0.8,
    eos_token_id=tok.eos_token_id,
)
full = tok.decode(out[0], skip_special_tokens=True)
completion = full[len(prompt):]
print("COMPLETION:", completion)

Common Misconception

“The completion includes the prompt.” APIs often return the full string or only new tokens depending on settings—always know which. Logging and cost accounting should separate prompt vs. completion tokens.

Strengths and Tradeoffs

Strengths

Flexible natural-language outputs.
Streaming improves perceived latency.
Same loop for many apps.

Tradeoffs

Nondeterminism complicates tests.
Truncation yields partial JSON/code.
Must validate before side effects.

Knowledge Check

Short Answer: What is a completion? Answer: The model-generated continuation after the prompt.
True/False: max tokens can truncate mid-sentence. Answer: True.
Multiple Choice: Stop sequences: (a) end generation early on match, (b) train the tokenizer, (c) remove GPUs. Answer: (a).
Short Answer: Why separate prompt vs. completion tokens? Answer: Billing, logging, and clear I/O boundaries.
True/False: Greedy decoding always maximizes diversity. Answer: False.
Multiple Choice: Streaming completions: (a) send tokens as produced, (b) require offline batch only, (c) disable softmax. Answer: (a).
Short Answer: Name one Module 11.1 knob affecting completions. Answer: Temperature, top-k, or top-p.
Short Answer: What module comes next after this lecture? Answer: Module 11.4 (Large Language Model).
Multiple Choice: Before executing tool calls from a completion you should: (a) validate/parse, (b) trust blindly always, (c) delete logs. Answer: (a).
True/False: Chat assistant messages are a form of completion. Answer: True.

Key Takeaways

Completions are decoded token sequences conditioned on prompts.
Control length, stops, and sampling carefully.
Separate prompt/completion for ops and cost.
Validate outputs—especially structured or tool-bearing text.
Next module: Large Language Model.

Trainer’s Guide

Hands-on idea: Force JSON output via prompt; measure parse success under greedy vs. high temperature.

Discussion prompt: How should UIs present streamed partial completions safely?

Recap: Completions are the GPT output surface. Continue to Module 11.4 with Large Language Model.