DALL·E and CLIP
Era 4 · Deep learning and foundation models (2012–2021) › Generative image models
- Lineage
- Neurons and networks
- Kind
- Implementation
Who
OpenAI
What happened
On January 5, 2021, OpenAI presents DALL·E, a 12-billion-parameter Transformer that generates images from a written description, and CLIP, a model that connects images and text and is used to rank DALL·E's results.
Why it mattered
One model draws what it is asked for in natural language, and another understands images from their descriptions without hand-made labels. Together they bring vision and language together.
The full story
DALL·E, whose name blends the painter Salvador Dalí and Pixar's robot WALL·E, applied the GPT-3 recipe to images. A 12-billion-parameter Transformer treated text and image as a single sequence of tokens and learned to predict the next one. According to its paper, this simple approach, with enough data and scale, was competitive with systems designed for specific tasks without being trained for them. The image of an armchair in the shape of an avocado went around the world.
CLIP solved another problem. Vision systems recognized only a fixed list of categories, learned from hand-made labels. CLIP learned from 400 million image-text pairs collected from the internet, matching each image with its caption. Afterward it could recognize concepts described in words. Tested on more than 30 datasets, it matched the ImageNet accuracy of the original ResNet-50 without using any of its 1.28 million training images.
The two pieces were combined: CLIP ranked the images DALL·E generated. The papers with the details came out on arXiv in February.
In April 2022 DALL·E 2 swapped the Transformer for a diffusion model guided by CLIP's representations, and text-to-image generation reached the general public.
Builds on
Variational autoencoder (December 20, 2013)
Documented direct influence
DALL·E turns each image into tokens with a discrete variational autoencoder (dVAE), its first training stage (section 2)
Leads to
DALL·E 2 (April 6, 2022)
Documented direct influence
DALL·E 2 first generates an image representation in CLIP's space and then generates the image from it (abstract)
Threads that pass through here
- From diffusion to world models (step 3 of 7)
- From noise to image (step 5 of 7)
Sources
Original source
- Zero-Shot Text-to-Image Generation (arXiv:2102.12092) (opens in a new tab). Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever, 2021. open license: CC BY 4.0
- Learning Transferable Visual Models From Natural Language Supervision (arXiv:2103.00020) (opens in a new tab). Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al., 2021
Further reading
How to cite
“DALL·E and CLIP.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/dall-e-and-clip
This entry was last reviewed on November 13, 2026.