Mixtral 8x7B
Era 5 · Generative AI, reasoning, and agents (2022–2026) › Frontier models
- Lineage
- Neurons and networks
- Kind
- Implementation
Who
Mistral AI
What happened
The French company Mistral AI releases Mixtral 8x7B, a sparse mixture-of-experts model with open weights under the Apache 2.0 license. Of its 47 billion parameters, it uses only about 13 billion per token, and it matches or beats Llama 2 70B and GPT-3.5 on most benchmarks.
Why it mattered
It brings mixture of experts to open models: the capability of a large model at the running cost of a medium-sized one. It is Europe's main contribution to the language model race.
The full story
Mistral AI was founded in Paris in 2023, and by the end of that year it was already competing with the big American labs by releasing its models with open weights, that is, with the numerical values of the trained model available to anyone who wants to run or fine-tune it.
Mixtral 8x7B arrived in an unusual way. On December 8, 2023, the company posted on X a link to an 87 GB file to download over BitTorrent, with no further explanation. The official presentation, with its Apache 2.0 license, came on December 11.
Its architecture is a sparse mixture of experts. Each layer has eight distinct computation blocks, the experts, and a router picks two of them for each token. The model has about 47 billion parameters in total, but at each step it uses only about 13 billion, so it responds with the speed and cost of a much smaller model.
According to the technical paper, published in January 2024, Mixtral matched or beat Llama 2 70B and GPT-3.5 on almost every benchmark, with clear advantages in mathematics, coding, and multilingual text. The version fine-tuned to follow instructions beat GPT-3.5 Turbo, Claude 2.1, and Gemini Pro in human preference comparisons.
Mixture of experts later became the standard architecture of large models, open and closed, from DeepSeek-V3 to Gemini 3.
A token enters the router, which has lines to eight experts. Two of them, highlighted, join a final weighted-sum box; the other six go unused.
Under the hood
In a Transformer, each layer has an attention block and a computation block, the feed-forward network. Mixtral keeps attention shared and replaces the feed-forward network with eight of them, the experts.
For each token, a router computes a score per expert and keeps the two highest. The output is the sum of those two experts, weighted by their normalized scores: y = Σ softmax(TopK(x·W))ᵢ · Eᵢ(x), with K = 2.
That is why the arithmetic is not eight times seven. The name 8x7B adds up to 56 billion, but attention and the other layers are shared, and the total is 46.7 billion. Each token passes through 12.9 billion. The paper found that the router does not divide topics among experts: it chooses more by the form of the text, such as the syntax of code.
Leads to
DeepSeek-V3 (December 26, 2024)
Inherited, with no proven direct link
Both are open sparse mixture-of-experts models that activate only part of their parameters for each token
Sources
Original source
- Mixtral of Experts (arXiv:2401.04088) (opens in a new tab). Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch et al., 2024
Scholarship and context
- Mixtral of experts (Mistral AI, December 11, 2023) (opens in a new tab). Mistral AI, 2023
- Mixtral of experts (Simon Willison’s Weblog, December 11, 2023) (opens in a new tab). Simon Willison, 2023
How to cite
“Mixtral 8x7B.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/mixtral
This entry was last reviewed on November 13, 2026.