The History of AI

From reward to alignment

From the idea that reward reinforces behavior to the technique used to fine-tune ChatGPT. Two lines run in parallel and meet in Sutton; two more meet in Ziegler's work.

  1. The law of effect (1898)

    Thorndike formulates the law of effect, that behavior followed by reward is reinforced. It is the conceptual root of reinforcement learning.

  2. Samuel's checkers program (1952)

    Meanwhile, at IBM, Samuel's checkers program learns by playing against itself. In 1959 Samuel coins the term "machine learning." This line comes from programming, not psychology.

  3. Dynamic Programming (1957)

    Meanwhile, Bellman formulates dynamic programming, a method for solving multistage decision problems when the environment is known. This line comes from optimal control.

  4. Temporal differences (August 1988)

    Sutton publishes temporal-difference learning. It builds on Samuel's work. The debt to Thorndike is conceptual, with no direct line between the authors.

  5. Q-learning (1989)

    A year later, Watkins introduces Q-learning in a PhD thesis, Learning from Delayed Rewards. It brings temporal differences together with optimal control.

  6. TD-Gammon (1992)

    Tesauro applies Sutton's algorithm directly: a network trained by playing against itself reaches expert level at backgammon.

  7. DQN (December 19, 2013)

    DQN is Q-learning with a deep network. It plays Atari from the pixels.

  8. Reinforcement learning from human preferences (June 12, 2017)

    Deep reinforcement learning works when the reward is well defined. Christiano and colleagues take it to tasks where it is not, with an agent that learns from human comparisons. This is the origin of RLHF.

  9. PPO (July 20, 2017)

    Meanwhile, the same year, Schulman and colleagues publish PPO, a stable reinforcement learning algorithm. It is the one later used to align language models.

  10. RLHF for language models (September 18, 2019)

    Ziegler and colleagues apply RLHF to language models. Here the two lines meet: the human comparisons follow Christiano's method, and the policy is trained with PPO.

  11. InstructGPT (March 4, 2022)

    InstructGPT follows Ziegler's methodology and applies it to GPT-3: the model learns to follow instructions.

  12. ChatGPT (November 30, 2022)

    ChatGPT is a sibling model of InstructGPT, trained with the same methods. It reaches 100 million monthly users in two months.

Open in the interactive timeline