The MMDiT Architecture
Stable Diffusion 3 Medium abandons the U-Net backbone of SD 1.x and SDXL in favor of a Multimodal Diffusion Transformer (MMDiT). The key innovation: image patches and text tokens flow through the same transformer blocks with separate weight sets, allowing bidirectional information exchange. Images condition on text, but text representations also adapt to image content during the diffusion process.
This architecture change, detailed in the Stability AI announcement, explains the most visible improvement in SD3 Medium: text rendering. Previous SD models struggled to reliably render words inside images because language and vision were coupled late in the pipeline. With MMDiT, text tokens are present throughout denoising.
Three Text Encoders
SD3 Medium uses three text encoders simultaneously:
- CLIP-L (77 token limit) - captures broad semantic meaning
- CLIP-G (77 token limit) - higher-capacity CLIP variant for style/composition
- T5-XXL (512 token limit) - captures detailed, structured language understanding
All three embeddings are concatenated and passed to the MMDiT blocks. In practice, T5-XXL alone can be dropped to save ~10GB VRAM with minimal quality loss - useful for consumer hardware. The HuggingFace model page documents this tradeoff.
