Stable Diffusion 1.x/2.x made open stills practical at ~512². SDXL (Stable Diffusion XL, Stability AI, 2023) is the next catalog SKU: larger UNet, two text encoders, native ~1024², optional refiner. Same open-weight philosophy, heavier VRAM, better default composition.
Volume 17 returns to SDXL as an architecture (17.1 SDXL, DDPM, ControlNet, ComfyUI). Here you only need to select XL vs 1.5 vs FLUX vs closed APIs. Do not memorize fake “XL always beats FLUX” scores—spike on your brief, GPU, and license.
Learning Objectives
By the end of this lesson, students should be able to:
- Explain SDXL as dual-encoder latent diffusion at native ~1024 with optional refiner.
- Compare SDXL to SD 1.5 on VRAM, fidelity, text, and ecosystem maturity.
- Run a base (and optionally refiner)
diffuserspipeline. - Know OpenRAIL++-M style licensing vs FLUX schnell Apache vs closed ToS.
- Decide base-only vs base+refiner vs Turbo/Lightning distilled XL variants (qualitatively).
- Point to Vol. 17 for ControlNet-on-XL and ComfyUI graphs.
SDXL is Stability AI’s XL latent-diffusion stack: a larger denoiser plus CLIP ViT-L and OpenCLIP ViT-bigG text encoders (concatenated conditioning), trained for higher native resolution than SD 1.5. A separate refiner checkpoint can take over in the last denoising steps for high-frequency detail. Distilled cousins (SDXL Turbo, Lightning, community speed LoRAs) trade steps for latency—same family, different ops profile.
Rubric: SD 1.5 vs SDXL vs FLUX
| Axis | SD 1.5 | SDXL | FLUX (preview) |
|---|---|---|---|
| Open vs closed | Open weights (RAIL-M) | Open weights (typically OpenRAIL++-M on official base) | Mixed: schnell Apache; dev restricted; pro API |
| Prompt fidelity | Craft-heavy | Better default composition / prompt use than 1.5 | Often stronger literal + layout than XL |
| Text rendering | Weak | Still weak vs Ideogram / FLUX; better than 1.5 slightly, not solved | Notably stronger among open-ish still models |
| Licensing / commercial | RAIL-M restrictions | OpenRAIL++-M—read the card | Per-SKU: Apache vs non-comm vs paid API |
| Latency / VRAM | Lighter; 8 GB class possible | Heavier base; refiner adds another pass; Turbo/Lightning cut steps | Transformer/flow models; VRAM hungry unless schnell/quantized |
Base, Refiner, and Distilled XL
Base only
- One pipeline, simpler serving
- Good default for APIs
- Most LoRAs target base
Base + refiner
- Hand off last steps to refiner
- More detail, more VRAM/time
- ComfyUI makes the split explicit
Turbo / Lightning
- Few-step sampling
- Interactive look-dev
- Quality/diversity trade-off
Minimal SDXL diffusers Call
Official base id is commonly stabilityai/stable-diffusion-xl-base-1.0. Refiner is optional. Confirm revision/variant tags on the model card. Vol. 17 will explain why dual encoders help; here you just pass one prompt string (diffusers tokenizes for both).
Ecosystem and Vol. 17 Pointers
SDXL inherited the SD culture: LoRAs, ControlNet ports, IP-Adapter, A1111, ComfyUI. Some 1.5 LoRAs do not apply to XL—dimension mismatch. ControlNet weights are XL-specific. If the team’s library is 1.5-only, migrating is a project, not a flag. Preview LoRA (Diffusion) and ControlNet before promising pose-locked XL in production.
Pick SDXL when
- You want open 1024-class stills
- Existing ComfyUI/XL LoRA library
- 1.5 upscalers feel like a hack
- You can spare the VRAM
Move on when
- Text/layout is the pain → FLUX or Ideogram
- 8 GB laptop is the target → 1.5 or distilled
- Zero ops → DALL·E / Imagen / Firefly
- You need flow-matching theory → Vol. 17 + FLUX lecture
“SDXL always needs the refiner, and Turbo is just XL with a different name.” Many production APIs ship base only. The refiner is an extra model and a scheduling choice. Turbo/Lightning are distilled few-step variants—different training, different guidance behavior, not a drop-in quality upgrade. Choose on latency budget, then verify look on your prompts.
Knowledge Check
- Short Answer: What is distinctive about SDXL text conditioning vs SD 1.5? Answer: Two text encoders (CLIP L + OpenCLIP bigG), concatenated.
- True/False: SDXL native resolution is the same ~512² as SD 1.5. Answer: False—XL is native ~1024-class.
- Multiple Choice: The refiner is: (a) a Discord bot, (b) an optional second checkpoint for late denoising, (c) a video codec. Answer: (b).
- Short Answer: Why might a 1.5 LoRA fail on XL? Answer: Architecture/dim mismatch—adapters are checkpoint-family specific.
- True/False: Official SDXL base is typically closed like DALL·E. Answer: False—official base weights are openly downloadable (OpenRAIL++-M style terms).
- Multiple Choice: Best next lecture for stronger open text/layout: (a) FLUX, (b) Canva, (c) Whisper. Answer: (a).
- Short Answer: Name one Vol. 17 topic you need for pose-locked XL. Answer: ControlNet (also ComfyUI graphs).
- True/False: SDXL Turbo is just more inference steps on the same base. Answer: False—it is a distilled few-step variant with different behavior.
- Multiple Choice: XL vs 1.5 VRAM is typically: (a) lighter on XL, (b) heavier on XL base (+ refiner worse), (c) identical. Answer: (b).
- Short Answer: When is base-only SDXL enough? Answer: Most API serving; add refiner only if the extra pass earns its latency/VRAM.
Key Takeaways
- SDXL = larger latent diffusion + dual text encoders + ~1024 native, optional refiner.
- OpenRAIL++-M still has use restrictions—read the card.
- 1.5 LoRAs/ControlNets do not magically port; plan the migration.
- Vol. 17 SDXL / DDPM / ControlNet / ComfyUI is the deep path.
- Continue with FLUX.
Lab: Same prompt on SD 1.5 vs SDXL base (diffusers or ComfyUI). Compare VRAM, wall time, and whether small text in the scene is readable. No invented metrics.
Whiteboard: Dual-encoder stick figure. Optional refiner as a second UNet taking over late sigmas. Arrow to FLUX as “next open-ish leap.”
Recap: SDXL is the open 1024-class SD generation; refiner and Turbo are options, not obligations. Continue with FLUX.