ComfyUI — Module 17.1

How This Lesson Fits the Module & Volume

Every idea in 17.1—samplers, SD / SDXL / FLUX, ControlNet, LoRA, DreamBooth, inpaint / outpaint—has to be wired for real work. ComfyUI is the node-graph studio and a common production runner: JSON workflows, queue, API. Volume 16 pointed here from the catalog; this lecture is the graph itself.

The capstone sibling is Automatic1111: form-based Gradio UI, faster to click, harder to version as a DAG. After this volume, Vol. 18 OpenAI SDK and deployment (FastAPI, Docker, GPUs) is how you wrap either UI behind a product.

Learning Objectives

By the end of this lesson, students should be able to:

Describe ComfyUI as a node graph: models, CLIP encode, sampler, VAE, save.
Contrast graph workflows vs Automatic1111’s tabbed forms.
Sketch txt2img, ControlNet, LoRA, and inpaint graphs without inventing fake node names as gospel.
Treat workflow JSON as the source of truth (git, review, API replay).
Plan production: queue, custom nodes, FLUX quant, reproducibility (seed + graph).
Know when A1111 is faster to teach and when ComfyUI is safer to ship.

Definition

ComfyUI is a graph-based interface and execution engine for diffusion (and related) pipelines. Each node is a typed step—load checkpoint, encode prompt, apply LoRA, run KSampler, VAE decode, save image. Edges carry tensors and conditioning. A workflow is serializable JSON you can diff, share, and POST to the ComfyUI server. That is the opposite of a single opaque “Generate” button whose hidden extras live only in PNG metadata.

Canonical txt2img Graph

CheckpointLoader

UNet/DiT + CLIP/T5 + VAE

CLIP Text Encode

Positive / negative prompts

Empty Latent

Size / batch

KSampler

Steps, cfg, sampler, seed

VAE Decode → Save

Pixels out

Concern	ComfyUI	Automatic1111
Mental model	Explicit DAG of tensors	Tabs + extra networks + extensions
Repro artifact	Workflow JSON (+ models)	PNG info / infotext; harder DAG
FLUX / weird graphs	First-class; custom nodes	Possible via forks/extensions; clunkier
Learning curve	Steeper day one	Faster for txt2img/inpaint demos
Production API	Native queue + prompt JSON	HTTP API exists; graphs still implicit
ControlNet / LoRA	Visible nodes and weights	Units / extra networks UI

Workflow JSON (pattern, not a full dump)

Real Comfy graphs are large numeric node ids. Teams version the JSON beside model hashes. The API accepts a prompt mapping node id → class_type + inputs, then you poll the queue. Do not hard-code node numbers from a screenshot into prod without exporting the actual file.

# Conceptual ComfyUI API payload (ids simplified)
prompt = {
  "1": {
    "class_type": "CheckpointLoaderSimple",
    "inputs": {"ckpt_name": "sd_xl_base_1.0.safetensors"},
  },
  "2": {
    "class_type": "CLIPTextEncode",
    "inputs": {"text": "oak chair, soft studio light", "clip": ["1", 1]},
  },
  "3": {
    "class_type": "CLIPTextEncode",
    "inputs": {"text": "watermark, extra legs", "clip": ["1", 1]},
  },
  "4": {
    "class_type": "EmptyLatentImage",
    "inputs": {"width": 1024, "height": 1024, "batch_size": 1},
  },
  "5": {
    "class_type": "KSampler",
    "inputs": {
      "seed": 42, "steps": 28, "cfg": 6.5,
      "sampler_name": "dpmpp_2m", "scheduler": "karras",
      "model": ["1", 0], "positive": ["2", 0], "negative": ["3", 0],
      "latent_image": ["4", 0],
    },
  },
  "6": {"class_type": "VAEDecode", "inputs": {"samples": ["5", 0], "vae": ["1", 2]}},
  "7": {"class_type": "SaveImage", "inputs": {"images": ["6", 0], "filename_prefix": "xl_chair"}},
}
# POST /prompt  {"prompt": prompt}
# Production: pin ckpt/LoRA hashes; add LoRALoader / ControlNetApply / VAEEncode for img2img

Production Graphs You Will Actually Build

Control + LoRA

Load image → Canny/OpenPose preprocessor
ControlNetApply + LoraLoader
Same KSampler seed for A/B

Inpaint / outpaint

VAEEncode + mask
PadImage for outpaint
Inpaint model or masked sampler

FLUX serve

Quantized UNet/DiT loaders
Few-step sampler, guidance ~0
Offload nodes for 24 GB

When ComfyUI vs A1111

Pick ComfyUI when

Workflow must be reviewed in git
FLUX / multi-ControlNet / custom nodes
Queue + API is the product backend
Repro > click speed

Pick A1111 when

Teaching txt2img/inpaint in an hour
Huge extension zoo on SD 1.5
Designers live in sliders, not DAGs
PNG infotext is enough audit

Related Lectures

Lecture	Why it sits beside ComfyUI
Automatic1111	Form UI twin / volume capstone
FLUX / SDXL	Checkpoints you drop into loaders
ControlNet / LoRA	Nodes you insert on the conditioning path
Inpaint / Outpaint	Encode + mask + pad subgraphs
Vol. 16.3 FLUX	Catalog reminder: license before you graph [dev]
Vol. 18 FastAPI	Wrap the Comfy queue in a product API

Common Misconception

“ComfyUI is just a prettier Automatic1111.” It is a different contract: explicit tensors vs hidden pipeline. A second mistake: sharing a JSON without the checkpoint/LoRA hashes—the graph will load and silently use the wrong file. Third: treating custom nodes as core Comfy; pin versions or your prod queue breaks on update.

Knowledge Check

Short Answer: What artifact makes a ComfyUI run reviewable in git? Answer: The workflow JSON (plus pinned model/LoRA hashes).
True/False: ComfyUI executes a node DAG rather than a single opaque Generate button. Answer: True.
Multiple Choice: Typical sampler node family: (a) KSampler, (b) k-means++, (c) Celery beat. Answer: (a).
Short Answer: Name one reason teams prefer ComfyUI over A1111 in production. Answer: Explicit graphs, API/queue, FLUX/custom-node flexibility, or reproducibility.
True/False: A1111 is usually slower to demo txt2img to beginners. Answer: False—A1111 tabs are often faster to click on day one.
Multiple Choice: Outpaint in Comfy is typically: (a) Pad + inpaint/mask nodes, (b) a new NLP tokenizer, (c) PCA. Answer: (a).
Short Answer: What should you pin besides the JSON? Answer: Checkpoint / LoRA / VAE / custom-node versions (hashes).
True/False: FLUX schnell graphs should copy SDXL’s 30-step CFG 7.5 defaults. Answer: False—few steps, low/zero guidance.
Multiple Choice: Volume capstone UI sibling: (a) Automatic1111, (b) Word2Vec, (c) Mixtral. Answer: (a).
Short Answer: Which volume wraps UIs behind SDKs and deploy? Answer: Vol. 18 (OpenAI SDK, FastAPI, Docker, GPUs).

Key Takeaways

ComfyUI is a typed node graph + JSON workflow + queue API.
Make tensors visible: checkpoint, CLIP, LoRA, ControlNet, sampler, VAE.
Prefer Comfy for production repro; A1111 for fast UI teaching.
Pin models and custom nodes or the graph lies.
Continue with Automatic1111 (volume capstone).

Trainer’s Guide

Lab: Recreate txt2img, then add LoRA + Canny ControlNet as extra nodes. Export JSON, break a model filename, watch the queue fail. Compare the same brief in A1111.

Whiteboard: Draw SDXL graph vs FLUX few-step graph. Box “JSON = source of truth.” Arrow to Vol. 18 FastAPI wrapping /prompt.

Recap: ComfyUI turns diffusion into a versioned node graph. Continue with Automatic1111.