Together AI — Module 22.5

How This Lesson Fits the Module & Volume

Hugging Face distributes weights; Replicate runs Cog jobs. Together AI is an open-model inference and fine-tuning cloud: Llama, Qwen, Mixtral-class SKUs over an OpenAI-compatible API, plus dedicated endpoints and training. Vol. 11.5 Llama / Qwen / Mixtral are the families; Together is one place you rent GPUs already wired to those chat templates.

Speed specialists next: Groq and Cerebras. Gateway: OpenRouter. Client code looks like Vol. 18 OpenAI SDK with a different base_url.

Learning Objectives

By the end of this lesson, students should be able to:

Describe Together as hosted open-weight LLM inference + fine-tune, not a Hub.
Call Together with the OpenAI Python SDK via base_url and env key.
Compare Together vs Groq vs HF Endpoints vs frontier APIs qualitatively.
Know serverless vs dedicated endpoint trade-offs (noisy neighbor vs commit).
Treat model IDs as SKUs; keep chat templates compatible with the family (Vol. 11.5).
Decide when Together fine-tune beats prompt+RAG (Vol. 14) for a stable task.

Definition

Together AI is a cloud inference (and fine-tuning) provider for open and some partner models. You authenticate with an API key, select a model id (e.g. a Llama or Qwen instruct checkpoint—names evolve), and call chat/completions or embeddings over HTTPS. Dedicated endpoints pin capacity. Together does not replace your RAG store, your FastAPI auth, or Hugging Face licenses on the underlying weights.

Open Weights, Hosted Runtime

Buying Together is not “we trained Llama.” You still obey the weight license (Llama Community, Apache, Qwen license, …). You buy ops: batching, scaling, often OpenAI wire compatibility so Vol. 18 code barely changes. Frontier vendors (Vol. 22.1) still win many closed-model evals; Together wins when you want open weights + someone else’s GPU fleet.

Pricing / Strengths / Weaknesses (Qualitative)

Dimension	Together AI	Groq	HF Endpoints	Frontier API (OpenAI/Anthropic/…)
Pricing posture	Per-token inference; dedicated hardware commits; fine-tune jobs extra—confirm live table	Per-token on LPU catalog	Instance-hours or serverless requests	Per-token closed SKUs
Strengths	Broad open-model catalog; OpenAI-compatible; fine-tune + inference one vendor; embeddings too	Very high decode speed on supported models	Same ids as Hub; custom containers	Strongest closed models; tool ecosystems
Weaknesses	Not the single fastest chip; serverless contention possible; you still eval quality; license != Together ToS	Narrower model list (must fit LPU)	You size GPUs yourself more often	Closed weights; data-handling review

Serverless inference

Fast to start
Pay per token
Watch noisy-neighbor latency

Dedicated endpoint

Predictable capacity
Commit cost even when idle
Better for SLOs

Fine-tune

Stable style/format tasks
Not a policy oracle (Vol. 20)
Still retrieve facts (Vol. 14 RAG)

Pick Together when

You want Llama/Qwen/Mistral-class APIs
OpenAI SDK already in the repo
You may fine-tune later on same vendor
Frontier quality is not required

Look elsewhere when

You need Groq/Cerebras-class decode
You need Claude/GPT-class closed quality
You must air-gap (self-host)
Workload is diffusion video (Replicate)

Python: OpenAI SDK, Together Base URL

This is the Vol. 18 pattern: one client class, swap base_url + key + model id. Pin the Together model string in config. Confirm current compatibility path in Together docs (OpenAI vs native SDK).

# together_chat.py — OpenAI-compatible client (Vol. 18 SDK pattern)
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["TOGETHER_API_KEY"],
    base_url="https://api.together.xyz/v1",
)
resp = client.chat.completions.create(
    model=os.environ.get("TOGETHER_MODEL", "meta-llama/Llama-3.3-70B-Instruct-Turbo"),
    messages=[
        {"role": "system", "content": "Be concise. You are not the authorization oracle."},
        {"role": "user", "content": "In one sentence, what is Together AI?"},
    ],
    max_tokens=128,
)
print(resp.choices[0].message.content)
print(resp.usage)  # meter tokens in your Vol. 19 token-usage logs

Related Lectures

Lecture	Role
Llama / Qwen / Mistral	Weight families
OpenAI SDK / FastAPI	Client + wrap
RAG	Facts still retrieved, not fine-tuned blindly
Groq / OpenRouter	Speed / multi-provider next
Token usage / latency	Eval meters

Common Misconception

“Together is Llama.” Together hosts many families. Second: OpenAI-compatible means GPT-4 quality. Third: fine-tune replaces RAG for weekly FAQs. Fourth: serverless dedicated-grade SLO without measurement. Fifth: Together ToS overrides the Llama license. Sixth: Groq and Together are the same hardware.

Knowledge Check

Short Answer: What does Together primarily sell? Answer: Hosted open-model inference (and fine-tuning), via API.
True/False: Switching base_url to Together makes the model GPT-class automatically. Answer: False.
Multiple Choice: Weight licenses: (a) still apply when Together hosts the model, (b) vanish, (c) are replaced by Suno ToS. Answer: (a).
Short Answer: Name one qualitative strength vs Groq. Answer: Broader model catalog / fine-tune offering / embeddings (any valid).
True/False: Dedicated endpoints trade idle cost for more predictable capacity. Answer: True.
Multiple Choice: Client pattern is closest to: (a) Vol. 18 OpenAI SDK, (b) Suno consumer UI only, (c) Cartesia cloning studio. Answer: (a).
Short Answer: Which Vol. 11.5 family is a common Together SKU? Answer: Llama, Qwen, Mistral/Mixtral, or similar open family.
True/False: Invent per-million-token prices from this page for finance sign-off. Answer: False.
Multiple Choice: Changing weekly facts should usually use: (a) RAG, (b) only fine-tune, (c) Replicate FLUX. Answer: (a).
Short Answer: Next speed-specialist lecture? Answer: Groq.

Key Takeaways

Together = open-weight inference + optional fine-tune on an OpenAI-shaped API.
Licenses and eval still belong to you; Together is the runtime.
Serverless vs dedicated is an SLO/cost choice, not a quality magic trick.
Swap base_url in Vol. 18 clients; pin model ids.
Next: Groq—LPU inference speed.

Trainer’s Guide

Lab: Same 10-prompt eval set through Together (or mocked OpenAI-compatible server) vs a frontier mini model if keys exist. Students log latency and qualitative quality—no invented tokens/sec leaderboards. Grade: env keys, model id in config, one paragraph on dedicated vs serverless for a voice-agent TTFT budget (link Vol. 21 voice + Vol. 19 latency).

Recap: Together hosts open LLMs with a familiar SDK shape. Measure quality and latency yourself. Hardware speed next: Groq.