← Master Index
Vol. 17 Module 17.1 Lecture

FLUX

Diffusion Foundations

How This Lesson Fits the Module & Volume

Module 17.1 built the physics: diffusion, noise, denoising, latent space, DDPM, and DDIM. Then came the classic UNet family: Stable Diffusion and SDXL.

FLUX is the modern DiT-style generator you actually ship in 2024+. Volume 16 already covered the catalog view—SKUs, licenses, when to buy FLUX vs SDXL vs DALL·E. This lecture opens the architecture: multimodal diffusion transformer, flow matching, and why it is not “a bigger SDXL UNet.” Control, adapters, and UIs follow: ControlNet, LoRA (Diffusion), ComfyUI.

Learning Objectives

By the end of this lesson, students should be able to:

  • Contrast FLUX (DiT / flow-matching) with SD 1.5/SDXL (UNet / DDPM-style latent diffusion).
  • Name the FLUX.1 SKUs (schnell, dev, pro) and what each license allows.
  • Run a schnell diffusers call with few steps and low/zero guidance.
  • Explain why prompt following and short text are stronger than classic SDXL, while VRAM is hungrier.
  • Point back to the Vol. 16 catalog lecture for purchase/license decisions.
  • Know where ControlNet-like control, LoRA, and ComfyUI graphs attach next.
Definition

FLUX.1 (Black Forest Labs) is a family of text-to-image generators built as a multimodal diffusion transformer (DiT / MMDiT-style) trained with flow matching (rectified-flow) rather than a classic UNet + discrete DDPM. Product-wise it ships as schnell (fast, Apache-2.0), dev (downloadable weights, non-commercial unless BFL licenses you), and pro (closed API). Later BFL still SKUs may appear—re-read the card; the transformer + flow + three-way license pattern is the durable lesson.

DiT Generator vs Stable Diffusion

SD 1.5 and SDXL denoise a VAE latent with a UNet conditioned on CLIP (and OpenCLIP on XL). FLUX still works in a compressed latent, but the denoiser is a transformer that jointly attends to text tokens and image patches. The training objective is closer to learning a velocity field along a straight path from noise to data (flow matching) than to predicting ε at discrete DDPM timesteps. That is why schnell can look good in ~1–8 steps, and why you must not copy SDXL’s 30 steps @ CFG 7.5 blindly.

AxisSD 1.5 / SDXLFLUX.1
DenoiserUNet (encoder–decoder + skips)Multimodal DiT / transformer blocks
Objective (typical)DDPM / v-prediction noiseFlow matching / rectified flow
Text encodersCLIP (XL: CLIP + OpenCLIP)T5-XXL + CLIP-L (confirm current card)
Native look1.5 ~512²; XL ~1024²High-res stills; strong layout/text
Steps / guidance20–50 steps, CFG ~5–8schnell: few steps, guidance ~0; dev: more steps
VRAM1.5 light; XL heavierFull BF16 is hungry—quant / offload / API
EcosystemHuge LoRA / ControlNet zooGrowing; ComfyUI-first; ports in progress

The Three SKUs (Architecture Meets License)

schnell

  • Distilled, few-step inference
  • Apache-2.0—commercially friendly
  • Slightly behind dev/pro on hard briefs

dev

  • Fuller quality among open weights
  • Weights ≠ commercial license
  • Startup foot-gun if shipped unpaid

pro

  • Closed BFL API
  • Best vendor stills, no GPU ops
  • ToS + paid usage

schnell via diffusers

Hugging Face id is commonly black-forest-labs/FLUX.1-schnell. Few steps and guidance_scale=0 (or very low) are typical for schnell. Confirm the current model card; FP8 / GGUF / NF4 quantization is how 24 GB cards survive, not a quality leaderboard. For license and catalog scoring, keep the Vol. 16.3 FLUX lecture open beside this one.

from diffusers import FluxPipeline import torch pipe = FluxPipeline.from_pretrained( "black-forest-labs/FLUX.1-schnell", torch_dtype=torch.bfloat16, ) pipe.enable_model_cpu_offload() # 24 GB-class survival tactic image = pipe( "a geometric lighthouse poster, fog, readable title LUMEN, no logos", num_inference_steps=4, guidance_scale=0.0, max_sequence_length=256, ).images[0] image.save("flux_schnell.png") # [dev] uses more steps and a non-Apache license—read the card before prod. # Quantized / GGUF / ComfyUI graphs are the usual self-host path.

When FLUX Beats SDXL—and When It Does Not

Pick FLUX when

  • Literal prompt following and short on-image text matter
  • schnell Apache serving is the legal win
  • You will invest in quant + ComfyUI
  • You want DALL·E-class fidelity without OpenAI lock-in

Stay on SD / SDXL when

  • Tiny GPU or a mature 1.5 LoRA / ControlNet museum
  • You need battle-tested inpaint/outpaint UNets today
  • Poster typography is the product → Ideogram
  • [dev] would silently enter a paid SaaS

Related Lectures

LectureWhy it sits beside FLUX
Vol. 16.3 FLUXCatalog / license / buy vs build—this page is architecture
SDXLUNet baseline you just left
DDPM / DDIMDiscrete noise schedules vs flow matching
ControlNetSpatial conditioning ports onto FLUX/SD
LoRA (Diffusion)Low-rank adapters on transformer weights
ComfyUIProduction FLUX graphs
Common Misconception

“FLUX is just SDXL with more parameters.” The denoiser class changed (transformer vs UNet) and so did the training objective (flow vs classic ε-DDPM). Copying SD sampler presets, CFG 7.5, and 50 steps onto schnell is a quality and latency bug. A second misconception: “weights on Hugging Face means Apache commercial use.” That is true for schnell, not for [dev]—same warning as Vol. 16.

Knowledge Check

  1. Short Answer: Name one architectural difference between FLUX and SDXL. Answer: FLUX uses a DiT/transformer denoiser + flow matching; SDXL uses a UNet + DDPM-style noise prediction.
  2. True/False: FLUX [dev] on Hugging Face is Apache-2.0 for commercial SaaS. Answer: False—dev is not Apache; commercial use needs a BFL license.
  3. Multiple Choice: schnell inference typically uses: (a) 50 DDPM steps @ CFG 12, (b) few steps and low/zero guidance, (c) Discord --stylize. Answer: (b).
  4. Short Answer: Which Vol. 16 lecture is the catalog/license twin of this page? Answer: Vol. 16.3 FLUX.
  5. True/False: schnell is usually the commercially simplest self-host SKU. Answer: True—Apache-2.0.
  6. Multiple Choice: Full BF16 FLUX VRAM is typically: (a) trivial on 8 GB, (b) a real constraint—quant/offload/API, (c) zero because it is CSS. Answer: (b).
  7. Short Answer: Why might a team still choose SD 1.5 over FLUX? Answer: Tiny GPU, huge LoRA/ControlNet zoo, or mature inpaint UNets.
  8. True/False: FLUX solves poster typesetting as well as Ideogram. Answer: False—better than SDXL, still not a layout specialist.
  9. Multiple Choice: Next control lecture in this module: (a) Whisper, (b) ControlNet, (c) MCP stdio. Answer: (b).
  10. Short Answer: Name the three FLUX.1 SKUs. Answer: schnell, dev, and pro.

Key Takeaways

  • FLUX is a DiT / flow-matching still generator, not a larger SDXL UNet.
  • schnell = Apache + few steps; dev = weights ≠ commercial; pro = API.
  • Vol. 16 teaches the purchase order; this lecture teaches the architecture.
  • VRAM, quant, and ComfyUI are part of choosing FLUX—not optional footnotes.
  • Continue with ControlNet for spatial conditioning.
Trainer’s Guide

Lab: Same prompt on SDXL (30 steps, CFG 7.5) and FLUX schnell (4 steps, guidance 0). Score readable signage and extra objects. If no GPU, compare published samples and discuss the license split with a fake “we shipped [dev] in prod” scenario.

Whiteboard: Two columns—UNet-DDPM vs DiT-flow. Arrow from Vol. 16 catalog FLUX into this page, then forward to ControlNet + ComfyUI.

Recap: FLUX is the modern transformer/flow still model—license the right SKU. Continue with ControlNet.