Some Moral and Technical Consequences of Automation
Era 2 · AI is born (1943–1979) › Cybernetics, information, and electronic computers
- Lineage
- Machines and computation
- Kind
- Publication
Who
MIT
What happened
Norbert Wiener publishes a paper in Science on the consequences of machines that learn and act faster than a person can supervise.
Why it mattered
It states what is now called the alignment problem: if a machine pursues a purpose we cannot correct in time, we had better be sure it is the purpose we truly want and not a colorful imitation of it.
The full story
By the late 1950s there were already programs that learned to play checkers better than the people who wrote them. Wiener started from that fact to take on a widespread idea: that a machine cannot do anything its programmer has not foreseen.
The answer is that learning machines can indeed surprise their makers, and that the danger grows when they act so fast that a person cannot tell, until it is too late, when to switch them off. To explain it, Wiener turns to tales of wishes granted all too literally, the genie of the Thousand and One Nights or W. W. Jacobs's "The Monkey's Paw," and to a concrete case: a machine trained on war games might pursue a nominal victory at the cost of the country's survival, if the rules of the game do not match what is really wanted.
Hence the paper's most quoted line: "If we use, to achieve our purposes, a mechanical agency with whose operation we cannot efficiently interfere once we have started it, … we had better be quite sure that the purpose put into the machine is the purpose which we really desire and not merely a colorful imitation of it."
The paper was one of the first public warnings about controlling intelligent machines. Today's debates on AI safety cite it as a forerunner of the problem of specifying a system's goals correctly.
Sources
Original source
Scholarship and context
How to cite
“Some Moral and Technical Consequences of Automation.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/wiener-consequences-of-automation
This entry was last reviewed on November 13, 2026.