← Master Index
Vol. 17 Module 17.1 Lecture

SDXL

Diffusion Foundations

How This Lesson Fits the Module & Volume

Stable Diffusion 1.x is the LDM contract at ~512 with one CLIP encoder. SDXL (Podell et al., 2023) is the same science scaled into a production SKU: larger UNet (~2.6B vs ~860M), two text encoders, native ~1024×1024, size/crop micro-conditioning, and an optional refiner. Volume 16.3’s SDXL catalog told you VRAM, RAIL++-M, and when to pick XL vs 1.5 vs FLUX. This lecture opens the tensors.

Downstream 17.1 topics—ControlNet, LoRA, ComfyUI—assume you can distinguish base vs refiner vs Turbo/Lightning distillation. FLUX next is a different denoiser family (flow + transformer), not “SDXL 3.”

Learning Objectives

By the end of this lesson, students should be able to:

  • Describe SDXL as dual-encoder latent diffusion at native ~1024 with optional refiner.
  • Name the two text encoders (CLIP ViT-L and OpenCLIP ViT-bigG) and how their embeddings are concatenated.
  • Explain micro-conditioning: original size, crop coords, target size as extra Fourier condition.
  • Run base-only and base→refiner pipelines in diffusers without mixing prediction types blindly.
  • Contrast XL vs 1.5 on latent grid, VRAM, CFG defaults, and ecosystem maturity.
  • Place Turbo / Lightning distillations as few-step cousins, not replacements for the XL training recipe.
Definition

SDXL (Stable Diffusion XL) is Stability AI’s 2023 LDM: a 3×-wider/deeper UNet denoiser operating on f=8, 4-channel latents at 128×128 for 1024×1024 RGB, conditioned on the concatenation of CLIP ViT-L and OpenCLIP ViT-bigG text embeddings plus pooled text vectors and micro-conditioning (canvas size, crop offset, target size). A separate refiner UNet, trained to specialize in high-noise-end / high-frequency detail, can take over for the last portion of denoising or run as img2img on the base output. Distilled variants (SDXL Turbo via adversarial diffusion distillation, Lightning, community speed LoRAs) keep the XL body and change the sampler budget.

What Actually Got Bigger

AxisSD 1.5SDXL base
Native pixels~512×512~1024×1024 (multi-aspect trained)
Latent grid (f=8, 4 ch)64×64128×128
UNet size~860M~2.6B
Text encodersCLIP ViT-L/14 onlyViT-L + OpenCLIP bigG (concat + pooled)
Extra condt + tokens+ size, crop, aesthetic-related training signals
Second stage—Optional refiner UNet
VRAM (fp16, ballpark)6–8 GB class possible10–12+ GB comfortable; refiner adds a pass

Dual Text Encoders and Micro-Conditioning

Token concat

  • L and bigG each emit token sequences.
  • Projected and concatenated on the channel axis.
  • Cross-attn K/V are wider than SD 1.5.

Pooled embeddings

  • Global text vectors added like class embeds.
  • Help prompt “gist” beyond token attention.
  • CFG still runs cond vs null on both paths.

Micro-conditioning

  • Original H×W, crop (top, left), target H×W.
  • Fourier-embedded, injected into the UNet.
  • Why XL handles non-square and “cropped LAION” better.

If you ignore micro-conditioning at inference (zeros / wrong size), XL can look like a randomly cropped 1.5 upscale: off-center subjects, weird framing. diffusers pipelines fill these from the output height/width you request—custom graphs in ComfyUI must pass them explicitly.

Base, Refiner, and Distillations

The base model does full denoising and is what most LoRAs target. The refiner is a second SDXL-sized UNet trained to operate in the later (higher-detail) portion of the schedule. Two serving patterns: (1) switch scheduler timestep from base to refiner at ~80% of steps (ensemble of experts), or (2) finish base, then refiner as img2img at low strength. Distilled XL (Turbo: adversarial diffusion distillation, ~1–4 steps; Lightning: few-step LoRA/full) is for latency. Do not evaluate Turbo with a 50-step DDIM CFG 7.5 recipe—wrong contract, same as mixing v-pred and ε-pred.

import torch from diffusers import StableDiffusionXLPipeline, StableDiffusionXLImg2ImgPipeline base = StableDiffusionXLPipeline.from_pretrained( "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16, variant="fp16", use_safetensors=True, ).to("cuda") prompt = "a brass telescope on a wooden desk, window light, no text" negative = "watermark, extra lenses, unreadable letters" latents_out = base( prompt=prompt, negative_prompt=negative, num_inference_steps=40, denoising_end=0.8, # stop early; hand off to refiner guidance_scale=5.0, # XL often wants lower CFG than 1.5 output_type="latent", generator=torch.Generator("cuda").manual_seed(0), ).images refiner = StableDiffusionXLImg2ImgPipeline.from_pretrained( "stabilityai/stable-diffusion-xl-refiner-1.0", torch_dtype=torch.float16, variant="fp16", use_safetensors=True, text_encoder_2=base.text_encoder_2, vae=base.vae, ).to("cuda") image = refiner( prompt=prompt, negative_prompt=negative, image=latents_out, num_inference_steps=40, denoising_start=0.8, guidance_scale=5.0, ).images[0] image.save("sdxl_base_refiner.png") # Base-only serving: omit refiner, denoising_end, output_type="pil" (default).

Ops and Product Choices (Now With Mechanism)

Choose XL when

  • You need open 1024 stills with better default composition than 1.5.
  • You will train XL LoRAs / ControlNets and can spend 12 GB+ VRAM.
  • Dual-encoder prompt following matters more than 1.5’s tiny-GPU story.

Stay on 1.5 or jump to FLUX when

  • 8 GB box / huge 1.5 adapter zoo → stay 1.5.
  • Literal prompt + text in frame → spike FLUX / Ideogram.
  • Zero GPU ops → Vol. 16 closed APIs (DALL·E, Imagen, Firefly).

Licensing remains OpenRAIL++-M style on official base weights—read the card, same warning as 16.3. Refiner and community fine-tunes can carry different terms. SVD is still SD-2.1-class temporal, not “SDXL video.”

Common Misconception

“SDXL is just SD 1.5 upscaled, the refiner is required, and XL always beats FLUX.” XL is retrained at larger latent resolution with new condition channels—not a 2× super-res pass on 1.5. Refiner is optional; many APIs ship base-only. FLUX is a flow-matching transformer with a three-SKU license split; it is not an SDXL checkpoint. Turbo/Lightning are few-step distillations: comparing them to 40-step base+refiner as if they were the same sampler is an invalid A/B.

Knowledge Check

  1. Short Answer: What are SDXL’s two text encoders? Answer: CLIP ViT-L and OpenCLIP ViT-bigG.
  2. True/False: SDXL UNets see 1024×1024 RGB directly. Answer: False—they see 4×128×128 latents (f=8).
  3. Multiple Choice: Micro-conditioning includes: (a) only CFG scale, (b) size / crop / target size embeddings, (c) the RAIL license text. Answer: (b).
  4. Short Answer: Rough UNet parameter counts for 1.5 vs XL base? Answer: ~860M vs ~2.6B.
  5. True/False: The refiner must run for SDXL to produce any valid image. Answer: False—base-only is a standard serving mode.
  6. Multiple Choice: XL CFG is often: (a) much higher than 1.5 (15–20), (b) similar or a bit lower (~4–7), (c) exactly 1.0 always. Answer: (b).
  7. Short Answer: What do denoising_end=0.8 on base and denoising_start=0.8 on refiner accomplish? Answer: Split the reverse schedule so refiner handles the last 20% of denoising.
  8. Short Answer: Name one few-step XL distillation family. Answer: SDXL Turbo (ADD) or Lightning (either is correct).
  9. Multiple Choice: Compared with Vol. 16 SDXL, this lecture adds: (a) only license tables, (b) dual-encoder / micro-cond / refiner mechanics, (c) Sora video training. Answer: (b).
  10. True/False: FLUX is an SDXL refiner checkpoint with a new name. Answer: False—it is a separate flow-matching transformer family.

Key Takeaways

  • SDXL scales the LDM contract: bigger UNet, 128×128 latents, dual text, micro-conditioning, optional refiner.
  • Pass size/crop/target correctly or XL framing falls apart—especially in Comfy graphs.
  • Base-only vs base+refiner vs Turbo/Lightning are different serving contracts on related weights.
  • Use Vol. 16 to pick XL vs 1.5 vs FLUX vs APIs; use this lecture to wire the tensors.
  • Next: FLUX—flow matching and the transformer generation stack.
Trainer’s Guide

Hands-on idea: Same prompt/seed: SD 1.5 @ 512, SDXL base-only @ 1024, SDXL base+refiner, SDXL Turbo @ 4 steps. Record VRAM (nvidia-smi), wall time, and a 1–5 composition score. No fake public leaderboards.

Discussion prompt: If micro-conditioning tells the UNet the original photo size used in training crops, what happens when every inference request lies and says 1024×1024 original with zero crop?

Recap: SDXL is dual-encoder latent diffusion at 1024 with optional refiner—same physics as 1.5, bigger contract. Continue with FLUX.