Stable Diffusion
Era 5 · Generative AI, reasoning, and agents (2022–2026) › Images, video, and the physical world
- Lineage
- Neurons and networks
- Kind
- Implementation
Who
CompVis (LMU Munich), Runway, Stability AI
What happened
Stable Diffusion is released, a text-to-image generator developed by the CompVis group at Ludwig Maximilian University of Munich and by Runway, with compute donated by Stability AI. Its code and weights are public, and it runs on ordinary computers.
Why it mattered
Latent diffusion with open weights. Anyone can run, modify, and fine-tune a high-quality image generator on their own computer.
The full story
Diffusion models generated excellent images, but they worked directly on pixels, and training and running them was very expensive. In December 2021 Robin Rombach, Björn Ommer, and colleagues proposed latent diffusion: first an autoencoder compresses images into a much smaller representation, and diffusion works in that reduced space. Cross-attention layers make it possible to guide generation with text. The method was presented at CVPR 2022 and greatly reduced the computation required.
Stable Diffusion applied that idea at scale. It was trained on image-text pairs from LAION-5B, a set of some five billion pairs collected from the web, and it was released on August 22, 2022, with its code and weights under the CreativeML OpenRAIL-M license. An optimized version ran with only 2.4 GB of graphics memory, within reach of many home computers.
Because it was open, anyone could run it, modify it, and fine-tune it to their own styles without going through a company. It also opened up conflicts: in January 2023 three artists sued Stability AI, Midjourney, and DeviantArt for copyright infringement, and Getty Images sued Stability AI in the United Kingdom.
SD 2.0, SDXL, and the SD 3 family followed.
A row of boxes, a 512-by-512 image, encoder, diffusion in a 64-by-64-by-4 space that removes noise, decoder, and image. From below, a box of text encoded by CLIP feeds into the diffusion through cross-attention. A dashed outline marks the latent space.
Under the hood
Latent diffusion separates two tasks. An autoencoder learns to compress an image and reconstruct it: a 512 × 512-pixel image with three color channels becomes a 64 × 64 grid with four channels, about 48 times fewer numbers.
The diffusion model, a U-Net, works only on that grid. It learns to remove noise step by step; to generate, it starts from pure noise, cleans it up over a few dozen steps, and hands the result to the decoder, which turns it into pixels.
Text comes in through cross-attention. A text encoder, CLIP's in the early versions, turns the sentence into vectors, and at each step the U-Net's layers consult them to decide what noise to remove.
Builds on
DDPM (June 19, 2020)
Documented direct influence
The DALL·E 2 decoder is a diffusion model, and Stable Diffusion's latent diffusion applies diffusion models in a compressed space; both papers cite Ho et al. (2020)
Variational autoencoder (December 20, 2013)
Documented direct influence
Latent diffusion works in the space of an autoencoder; one of its variants imposes a slight KL penalty, "similar to a VAE," citing Kingma and Welling
Leads to
Sora (February 15, 2024)
Inherited, with no proven direct link
Sora is a diffusion model that works on a compressed representation of video, split into patches like the images in the Vision Transformer
Threads that pass through here
- From diffusion to world models (step 5 of 7)
- From noise to image (step 7 of 7)
Sources
Original source
- High-Resolution Image Synthesis with Latent Diffusion Models (arXiv:2112.10752) (opens in a new tab). Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer, 2021
Further reading
How to cite
“Stable Diffusion.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/stable-diffusion
This entry was last reviewed on November 13, 2026.