The History of AI

DALL·E and CLIP

Era 4 · Deep learning and foundation models (2012–2021) › Generative image models

Lineage
Neurons and networks
Kind
Implementation

Who

OpenAI

What happened

On January 5, 2021, OpenAI presents DALL·E, a 12-billion-parameter Transformer that generates images from a written description, and CLIP, a model that connects images and text and is used to rank DALL·E's results.

Why it mattered

One model draws what it is asked for in natural language, and another understands images from their descriptions without hand-made labels. Together they bring vision and language together.

The full story

DALL·E, whose name blends the painter Salvador Dalí and Pixar's robot WALL·E, applied the GPT-3 recipe to images. A 12-billion-parameter Transformer treated text and image as a single sequence of tokens and learned to predict the next one. According to its paper, this simple approach, with enough data and scale, was competitive with systems designed for specific tasks without being trained for them. The image of an armchair in the shape of an avocado went around the world.

CLIP solved another problem. Vision systems recognized only a fixed list of categories, learned from hand-made labels. CLIP learned from 400 million image-text pairs collected from the internet, matching each image with its caption. Afterward it could recognize concepts described in words. Tested on more than 30 datasets, it matched the ImageNet accuracy of the original ResNet-50 without using any of its 1.28 million training images.

The two pieces were combined: CLIP ranked the images DALL·E generated. The papers with the details came out on arXiv in February.

In April 2022 DALL·E 2 swapped the Transformer for a diffusion model guided by CLIP's representations, and text-to-image generation reached the general public.

Builds on

  • Variational autoencoder (December 20, 2013)

    Documented direct influence

    DALL·E turns each image into tokens with a discrete variational autoencoder (dVAE), its first training stage (section 2)

Leads to

  • DALL·E 2 (April 6, 2022)

    Documented direct influence

    DALL·E 2 first generates an image representation in CLIP's space and then generates the image from it (abstract)

Threads that pass through here

Sources

Original source

How to cite

“DALL·E and CLIP.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/dall-e-and-clip

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline