The History of AI

Stable Diffusion

Era 5 · Generative AI, reasoning, and agents (2022–2026) › Images, video, and the physical world

Lineage
Neurons and networks
Kind
Implementation

Who

CompVis (LMU Munich), Runway, Stability AI

What happened

Stable Diffusion is released, a text-to-image generator developed by the CompVis group at Ludwig Maximilian University of Munich and by Runway, with compute donated by Stability AI. Its code and weights are public, and it runs on ordinary computers.

Why it mattered

Latent diffusion with open weights. Anyone can run, modify, and fine-tune a high-quality image generator on their own computer.

The full story

Diffusion models generated excellent images, but they worked directly on pixels, and training and running them was very expensive. In December 2021 Robin Rombach, Björn Ommer, and colleagues proposed latent diffusion: first an autoencoder compresses images into a much smaller representation, and diffusion works in that reduced space. Cross-attention layers make it possible to guide generation with text. The method was presented at CVPR 2022 and greatly reduced the computation required.

Stable Diffusion applied that idea at scale. It was trained on image-text pairs from LAION-5B, a set of some five billion pairs collected from the web, and it was released on August 22, 2022, with its code and weights under the CreativeML OpenRAIL-M license. An optimized version ran with only 2.4 GB of graphics memory, within reach of many home computers.

Because it was open, anyone could run it, modify it, and fine-tune it to their own styles without going through a company. It also opened up conflicts: in January 2023 three artists sued Stability AI, Midjourney, and DeviantArt for copyright infringement, and Getty Images sued Stability AI in the United Kingdom.

SD 2.0, SDXL, and the SD 3 family followed.

A row of boxes, a 512-by-512 image, encoder, diffusion in a 64-by-64-by-4 space that removes noise, decoder, and image. From below, a box of text encoded by CLIP feeds into the diffusion through cross-attention. A dashed outline marks the latent space.

Diffusion works on a 64 × 64 grid instead of the 512 × 512 pixels, and the text guides it at every step.

Under the hood

Latent diffusion separates two tasks. An autoencoder learns to compress an image and reconstruct it: a 512 × 512-pixel image with three color channels becomes a 64 × 64 grid with four channels, about 48 times fewer numbers.

The diffusion model, a U-Net, works only on that grid. It learns to remove noise step by step; to generate, it starts from pure noise, cleans it up over a few dozen steps, and hands the result to the decoder, which turns it into pixels.

Text comes in through cross-attention. A text encoder, CLIP's in the early versions, turns the sentence into vectors, and at each step the U-Net's layers consult them to decide what noise to remove.

Builds on

  • DDPM (June 19, 2020)

    Documented direct influence

    The DALL·E 2 decoder is a diffusion model, and Stable Diffusion's latent diffusion applies diffusion models in a compressed space; both papers cite Ho et al. (2020)

  • Variational autoencoder (December 20, 2013)

    Documented direct influence

    Latent diffusion works in the space of an autoencoder; one of its variants imposes a slight KL penalty, "similar to a VAE," citing Kingma and Welling

Leads to

  • Sora (February 15, 2024)

    Inherited, with no proven direct link

    Sora is a diffusion model that works on a compressed representation of video, split into patches like the images in the Vision Transformer

Threads that pass through here

Sources

Original source

How to cite

“Stable Diffusion.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/stable-diffusion

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline