Together is a broad open-model cloud. Groq is a hardware + inference company: Language Processing Units (LPUs) and GroqCloud APIs that decode supported LLMs extremely quickly. It is not xAI Grok (spelling). Not Cartesia. Not a Hub. Vol. 19 latency is why you care; Vol. 18 OpenAI SDK / Groq SDK is how you call it.
Gateway alternative: OpenRouter (often lists Groq as a provider). Wafer-scale peer: Cerebras. Voice agents (Vol. 21 / 22.4) love fast tokens if TTFT + TTS still fit the call budget.
Learning Objectives
By the end of this lesson, students should be able to:
- Distinguish Groq (LPU inference) from xAI Grok and from Together/HF.
- Call Groq via official SDK or OpenAI-compatible
base_url. - Explain why the model catalog is narrower (models must be compiled/ported to LPU).
- Compare qualitative pricing/strengths/weaknesses vs Together, Cerebras, frontier APIs.
- Measure tokens/sec and TTFT on your prompts—no fake speed tables.
- Know when speed does not fix a bad RAG or unsafe tool policy.
Groq designs LPUs (Language Processing Units) and operates GroqCloud: an API that serves a curated set of LLMs (Llama, Mixtral, and other ports—list changes) at very high decode rates. You pay per token on those SKUs. Groq is not a general GPU rental (not RunPod), not a diffusion host (not Replicate), and not the Grok chatbot.
Speed Is a Product Feature, Not a Quality Feature
LPUs optimize memory bandwidth and deterministic scheduling for transformer decode. Users feel snappy chat and better voice-agent turn times. The model is still the same open weight family—fast Llama is not Claude. Eval quality with Vol. 19 task metrics; eval speed with TTFT and tokens/sec on production-like prompts.
Pricing / Strengths / Weaknesses (Qualitative)
| Dimension | Groq | Together | Cerebras Inference | Frontier API |
|---|---|---|---|---|
| Pricing posture | Per-token on LPU-hosted models; confirm live GroqCloud table | Per-token GPU fleet; broader SKUs | Per-token wafer-scale inference | Per-token closed models |
| Strengths | Decode speed reputation; simple OpenAI-shaped API; great for interactive UX and some voice loops | Catalog + fine-tune | Also speed-specialist; different silicon story | Highest closed quality / tools |
| Weaknesses | Catalog limited to ported models; not vision/video supermarket; speed ≠ accuracy; capacity/SKU changes | Usually slower decode than LPU/WSE on comparable sizes (measure) | Also catalog-constrained | Often slower and costlier per token |
Use Groq
- Chat UX where wait hurts
- Agent loops with many small calls
- Voice LLM hop if model quality suffices
Use Together/HF
- Model not on Groq yet
- Fine-tune needed
- Embeddings / wider stack
Do not confuse
- Grok (xAI) vs Groq (LPU)
- Groq vs Cartesia TTS
- tok/s vs truthful answers
Do
- Pin model ids; log usage
- A/B quality vs a slower larger model
- Keep tools/HITL (Vol. 15/20)
- Read current model list before promising a SKU
Don’t
- Paste keynote tok/s into an SLO doc
- Assume every HF model runs on Groq
- Put Groq keys in frontend
- Skip wrap-as-data because it’s fast
Python: Groq SDK or OpenAI Compatible
Official groq package is thin. OpenAI client with Groq base URL also works—confirm current path. Model strings change; keep them in env.
Related Lectures
| Lecture | Role |
|---|---|
| xAI Grok | Different company (spelling trap) |
| Together / Cerebras / OpenRouter | Runtime peers / gateway |
| OpenAI SDK / streaming | Client + token stream |
| Latency / voice assistants | Why tok/s matters |
| Llama family | Typical Groq ports |
“Groq is Grok.” Different firms. Second: LPU speed means you can skip RAG and still be factual. Third: every open model on HF is available on GroqCloud. Fourth: published tok/s is your p95. Fifth: Groq replaces Cartesia TTS. Sixth: OpenRouter listing Groq means you don’t need data-processing agreements.
Knowledge Check
- Short Answer: What silicon story is Groq known for? Answer: LPUs (Language Processing Units) for fast LLM inference.
- True/False: Groq and xAI Grok are the same company. Answer: False.
- Multiple Choice: Groq’s catalog is narrower mainly because: (a) models must be ported to LPU, (b) they only do music, (c) they forbid OpenAI clients. Answer: (a).
- Short Answer: Name one qualitative strength vs Together. Answer: Decode speed / interactive latency (measure; any valid).
- True/False: Fast tokens guarantee better factuality. Answer: False.
- Multiple Choice: Client options include: (a) Groq SDK or OpenAI-compatible base URL, (b) only ComfyUI, (c) only Suno. Answer: (a).
- Short Answer: Which Vol. 19 lecture motivates measuring TTFT? Answer: Latency.
- True/False: You may invent tok/s leaderboards for this course. Answer: False.
- Multiple Choice: Voice agents still need: (a) STT/TTS vendors (Vol. 16/22.4), (b) only Groq, (c) Udio. Answer: (a).
- Short Answer: Which lecture is the multi-provider gateway next? Answer: OpenRouter.
Key Takeaways
- Groq = LPU inference cloud; not Grok, not TTS, not the Hub.
- Speed is real as a UX lever; quality is still the weight family + your eval.
- Catalog is curated; confirm SKUs before architecture lock-in.
- Same Vol. 18 client patterns; measure TTFT yourself.
- Next: OpenRouter—one key, many providers.
Lab: Same prompt streamed from Groq vs Together (or vs a local small model). Students record TTFT and qualitative answer quality. Grade: spelling Grok vs Groq correctly in the write-up, no copied marketing tok/s, env-based keys, note whether the Groq model id still exists on today’s dashboard.
Recap: Groq sells fast inference on a curated open-model list. Do not confuse it with Grok. Multi-provider routing next: OpenRouter.