Hugging Face is where weights live. Replicate is where many teams run someone else’s containerized model with a prediction API—especially image, video, and audio Cog packages that are painful to self-host. Vol. 16 image/video catalogs and Vol. 22.2/22.3 vendors often show up as Replicate model slugs. LLM-heavy serving more often moves to Together, Groq, or OpenRouter.
Wrap predictions behind Vol. 18 FastAPI + auth. Cog is packaging, not a substitute for Vol. 18 Docker literacy when you self-host later.
Learning Objectives
By the end of this lesson, students should be able to:
- Define Replicate as a hosted predictions platform (Cog containers + HTTP API), not a Model Hub.
- Call
replicate.run/ predictions with a pinned model version. - Contrast Replicate vs HF Inference vs Together vs self-host on price posture, cold start, and modality fit.
- Decide when GPU-second billing beats token billing (images/video vs chat).
- Treat publisher models as untrusted code+weights; read licenses.
- Isolate Replicate behind FastAPI so you can swap to HF Endpoints later.
Replicate is a cloud platform that runs machine-learning models as versioned containers (typically Cog: Docker + a predict function). You call a model owner/name and a version hash, pass input JSON/files, and receive output (URLs or bytes). Billing is generally per second of hardware used for that prediction, not per LLM token. Replicate is not Hugging Face Hub, not OpenRouter, and not your product auth layer.
When Predictions Beat Chat Completions
Chat vendors meter tokens. Diffusion, ASR fine-tunes, music separators, and video models meter GPU time. Replicate shines when the unit of work is a file in → file out. For Llama-class chat at high QPS, Together/Groq/Cerebras or self-hosted vLLM usually win the economics conversation—still verify with your traffic shape, not a blog.
Pricing / Strengths / Weaknesses (Qualitative)
| Dimension | Replicate | HF Inference / Endpoints | Together / Groq | Self-host Cog/vLLM |
|---|---|---|---|---|
| Pricing posture | GPU-second / prediction hardware SKU; idle scale-to-zero common—confirm live hardware prices | Per-request serverless or dedicated instance | Per-token LLM | Your GPU + eng |
| Strengths | Huge Cog catalog (vision/audio/video); version pins; simple Python client; great for spikes and prototypes | Same weights as Hub; tighter research loop | Fast cheap-ish LLM tokens (qualitative; measure) | Control, residency |
| Weaknesses | Cold starts on scale-to-zero; less ideal as sole LLM gateway; you trust the publisher’s container; costs spike on long video | Weaker “one-click obscure video model” catalog | Not a diffusion supermarket | You operate GPUs |
Good fit
- FLUX/SD-class stills
- Whisper variants, separators
- Occasional video jobs
- Hackathon demos
Weak fit
- Steady high-QPS chat
- Hard multi-tenant SLAs without dedicated hardware
- Air-gapped deploys
Always
- Pin version hashes
- Server-side token
- Log model+version in traces
Do
- Pin
owner/name:version - Timeout and retry idempotently
- Check model license + ToS
- Cache identical stills
Don’t
- Call Replicate from the browser with your token
- Leave version as “latest” in prod
- Assume output URLs live forever
- Invent $/second figures here
Python: Pinned Prediction
Model slugs change. The version hash is the real SKU. Download outputs server-side if you need retention; do not rely on ephemeral CDN URLs.
Related Lectures
| Lecture | Role |
|---|---|
| Hugging Face | Weights + cards |
| FLUX / Stable Diffusion / Whisper | Typical Cog workloads |
| Docker / FastAPI | Cog cousin + product wrap |
| Together AI | Next: token-metered open LLMs |
“Replicate is Hugging Face with a prettier client.” Different runtime and billing unit. Second: unpinned replicate.run("me/model") is reproducible. Third: output URLs are your backup strategy. Fourth: Replicate is cheaper than Groq for chat because GPU-seconds “feel small.” Fifth: Cog publishers are automatically license-clean.
Knowledge Check
- Short Answer: What unit does Replicate usually bill? Answer: Hardware time / GPU-seconds per prediction (not primarily tokens).
- True/False: Replicate is the Model Hub. Answer: False.
- Multiple Choice: Production calls should: (a) pin owner/name:version, (b) use latest, (c) embed the token in JS. Answer: (a).
- Short Answer: Name a modality Replicate fits well. Answer: Image, video, or audio file-in/file-out (any valid).
- True/False: Cold starts can happen on scale-to-zero hardware. Answer: True.
- Multiple Choice: High-QPS Llama chat often moves to: (a) Together/Groq/self-host, (b) only Replicate forever, (c) Suno. Answer: (a).
- Short Answer: What packaging format is associated with Replicate models? Answer: Cog (Docker + predict).
- True/False: You should invent $/GPU-second prices from this lecture. Answer: False.
- Multiple Choice: Vendor output URLs should be: (a) downloaded/persisted if you need them, (b) the only backup, (c) emailed as the API key. Answer: (a).
- Short Answer: Which Vol. 18 lecture wraps this HTTP into your app? Answer: FastAPI (or authentication / Docker).
Key Takeaways
- Replicate = versioned Cog predictions, GPU-second economics, great for media models.
- Pin versions; persist outputs; keep tokens server-side.
- Not automatically the right LLM host at high QPS.
- Read publisher licenses; wrap with Vol. 18.
- Next: Together AI—open-model token inference + fine-tune.
Lab: Students write a FastAPI route that accepts a prompt, calls a pinned Replicate model (or a mocked client), and stores the output URI in a local folder. Grade: version from env, no browser token, comment on cold-start UX, one sentence on when they would switch to HF Endpoints.
Recap: Replicate productizes file-in/file-out models. Pin, persist, wrap. Token-metered open LLMs next: Together AI.