The History of AI

Scale reaches China and Korea

Era 4 · Deep learning and foundation models (2012–2021) › Transformers and language models

Lineage
Neurons and networks
Kind
Implementation

Who

Huawei, Beijing Academy of Artificial Intelligence, Naver, Baidu

What happened

In 2021, labs in China and South Korea train language models as large as GPT-3 or larger: Huawei's PanGu-α, with up to 200 billion parameters (April 26), the Beijing Academy of Artificial Intelligence's Wu Dao 2.0, with 1.75 trillion (June), and Naver's HyperCLOVA, in Korean.

Why it mattered

The race for scale stops being the business of a few American labs, and large models are now trained in other languages and on homegrown hardware.

The full story

Baidu had already adapted BERT's recipe to Chinese in 2019 with ERNIE, which masked whole phrases and named entities during pretraining instead of single words, and improved the state of the art on five Chinese tasks. GPT-3, in 2020, changed the scale of the competition.

On April 26, 2021, a team from Huawei and other institutions presented PanGu-α, an autoregressive Chinese model with up to 200 billion parameters, trained on 1.1 TB of Chinese text on 2,048 of Huawei's own Ascend 910 processors, using its MindSpore framework.

In early June, the Beijing Academy of Artificial Intelligence (BAAI) unveiled Wu Dao 2.0 at its annual conference, with 1.75 trillion parameters, about ten times as many as GPT-3. It is a mixture-of-experts model: only part of the network activates for any given input. It wrote poems and essays in Chinese, described images, generated images from text, and predicted protein structures. BAAI's chairman, Zhang Hongjiang, said that the road to artificial general intelligence ran through big models and big computers.

In South Korea, Naver described HyperCLOVA in September, a GPT-3 variant with 82 billion parameters trained on 560 billion tokens of mostly Korean text, which set the state of the art in few-shot learning for that language.

Builds on

  • ELMo, GPT-1, and BERT (February 15, 2018), GPT-3 (May 28, 2020)

    Documented direct influence

    ERNIE draws on BERT's masking strategy, and PanGu-α and HyperCLOVA are presented as building on GPT-3 (the abstracts of all three papers)

Threads that pass through here

Sources

How to cite

“Scale reaches China and Korea.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/scale-reaches-china-and-korea

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline