Bias and accountability in AI
Era 4 · Deep learning and foundation models (2012–2021) › Alignment
- Lineage
- Data and learning
- Kind
- Publication
Who
MIT Media Lab, Microsoft Research, University of Washington, Google
What happened
In February 2018, in "Gender Shades," Joy Buolamwini and Timnit Gebru show that three commercial facial analysis systems fail far more often with darker-skinned women. In March 2021 Emily M. Bender, Gebru, and other coauthors warn of the risks of ever larger language models.
Why it mattered
Alongside the question of how to make a system do what is wanted, another one takes hold: whom it harms when it fails, and who answers for it.
The full story
Machine learning systems learn from the data they are trained on, and they inherit its gaps. Buolamwini, at the MIT Media Lab, and Gebru, at Microsoft Research, measured this in facial analysis. Two widely used benchmark datasets, IJB-A and Adience, were made up of 79.6 percent and 86.2 percent lighter-skinned people. Using a new dataset balanced by sex and skin type, they evaluated three commercial gender classification systems.
The result, presented at the FAT* conference in February 2018, was that darker-skinned women were the worst-classified group, with error rates of up to 34.7 percent, against a maximum of 0.8 percent for lighter-skinned men.
In 2020, by then at Google, Gebru wrote "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" with Emily M. Bender of the University of Washington and other coauthors. The paper pointed to the financial and environmental cost of training such models, to the hard-to-detect biases they absorb from huge amounts of text, and to the risk that they mislead, because they produce fluent language with no grounding in meaning.
In December 2020 Gebru left Google after the company asked for the paper to be withdrawn; Google said it was accepting Gebru's resignation, and Gebru said no resignation had been offered. The paper was published in March 2021 at the FAccT conference.
Sources
Original source
- Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Proceedings of FAT* 2018, PMLR 81, pp. 77-91 (opens in a new tab). Joy Buolamwini and Timnit Gebru, 2018
- On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? Proceedings of FAccT 2021, pp. 610-623 (opens in a new tab). Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Margaret Mitchell (signing as Shmargaret Shmitchell), 2021
How to cite
“Bias and accountability in AI.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/bias-and-accountability
This entry was last reviewed on November 13, 2026.