Scale reaches China and Korea
Era 4 · Deep learning and foundation models (2012–2021) › Transformers and language models
- Lineage
- Neurons and networks
- Kind
- Implementation
Who
Huawei, Beijing Academy of Artificial Intelligence, Naver, Baidu
What happened
In 2021, labs in China and South Korea train language models as large as GPT-3 or larger: Huawei's PanGu-α, with up to 200 billion parameters (April 26), the Beijing Academy of Artificial Intelligence's Wu Dao 2.0, with 1.75 trillion (June), and Naver's HyperCLOVA, in Korean.
Why it mattered
The race for scale stops being the business of a few American labs, and large models are now trained in other languages and on homegrown hardware.
The full story
Baidu had already adapted BERT's recipe to Chinese in 2019 with ERNIE, which masked whole phrases and named entities during pretraining instead of single words, and improved the state of the art on five Chinese tasks. GPT-3, in 2020, changed the scale of the competition.
On April 26, 2021, a team from Huawei and other institutions presented PanGu-α, an autoregressive Chinese model with up to 200 billion parameters, trained on 1.1 TB of Chinese text on 2,048 of Huawei's own Ascend 910 processors, using its MindSpore framework.
In early June, the Beijing Academy of Artificial Intelligence (BAAI) unveiled Wu Dao 2.0 at its annual conference, with 1.75 trillion parameters, about ten times as many as GPT-3. It is a mixture-of-experts model: only part of the network activates for any given input. It wrote poems and essays in Chinese, described images, generated images from text, and predicted protein structures. BAAI's chairman, Zhang Hongjiang, said that the road to artificial general intelligence ran through big models and big computers.
In South Korea, Naver described HyperCLOVA in September, a GPT-3 variant with 82 billion parameters trained on 560 billion tokens of mostly Korean text, which set the state of the art in few-shot learning for that language.
Builds on
ELMo, GPT-1, and BERT (February 15, 2018), GPT-3 (May 28, 2020)
Documented direct influence
ERNIE draws on BERT's masking strategy, and PanGu-α and HyperCLOVA are presented as building on GPT-3 (the abstracts of all three papers)
Threads that pass through here
- Scale as a method (step 6 of 8)
Sources
Original source
- PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation (arXiv:2104.12369) (opens in a new tab). Wei Zeng, Xiaozhe Ren, Teng Su, and 35 others, 2021
- What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA (arXiv:2109.04650) (opens in a new tab). Boseop Kim, HyoungSeok Kim, Sang-Woo Lee et al. (Naver), 2021
- ERNIE: Enhanced Representation through Knowledge Integration (arXiv:1904.09223) (opens in a new tab). Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng et al. (Baidu), 2019
Scholarship and context
Further reading
How to cite
“Scale reaches China and Korea.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/scale-reaches-china-and-korea
This entry was last reviewed on November 13, 2026.