The History of AI

Mixtral 8x7B

Era 5 · Generative AI, reasoning, and agents (2022–2026) › Frontier models

Lineage
Neurons and networks
Kind
Implementation

Who

Mistral AI

What happened

The French company Mistral AI releases Mixtral 8x7B, a sparse mixture-of-experts model with open weights under the Apache 2.0 license. Of its 47 billion parameters, it uses only about 13 billion per token, and it matches or beats Llama 2 70B and GPT-3.5 on most benchmarks.

Why it mattered

It brings mixture of experts to open models: the capability of a large model at the running cost of a medium-sized one. It is Europe's main contribution to the language model race.

The full story

Mistral AI was founded in Paris in 2023, and by the end of that year it was already competing with the big American labs by releasing its models with open weights, that is, with the numerical values of the trained model available to anyone who wants to run or fine-tune it.

Mixtral 8x7B arrived in an unusual way. On December 8, 2023, the company posted on X a link to an 87 GB file to download over BitTorrent, with no further explanation. The official presentation, with its Apache 2.0 license, came on December 11.

Its architecture is a sparse mixture of experts. Each layer has eight distinct computation blocks, the experts, and a router picks two of them for each token. The model has about 47 billion parameters in total, but at each step it uses only about 13 billion, so it responds with the speed and cost of a much smaller model.

According to the technical paper, published in January 2024, Mixtral matched or beat Llama 2 70B and GPT-3.5 on almost every benchmark, with clear advantages in mathematics, coding, and multilingual text. The version fine-tuned to follow instructions beat GPT-3.5 Turbo, Claude 2.1, and Gemini Pro in human preference comparisons.

Mixture of experts later became the standard architecture of large models, open and closed, from DeepSeek-V3 to Gemini 3.

A token enters the router, which has lines to eight experts. Two of them, highlighted, join a final weighted-sum box; the other six go unused.

For each token, the router picks two of the eight experts and adds up their outputs.

Under the hood

In a Transformer, each layer has an attention block and a computation block, the feed-forward network. Mixtral keeps attention shared and replaces the feed-forward network with eight of them, the experts.

For each token, a router computes a score per expert and keeps the two highest. The output is the sum of those two experts, weighted by their normalized scores: y = Σ softmax(TopK(x·W))ᵢ · Eᵢ(x), with K = 2.

That is why the arithmetic is not eight times seven. The name 8x7B adds up to 56 billion, but attention and the other layers are shared, and the total is 46.7 billion. Each token passes through 12.9 billion. The paper found that the router does not divide topics among experts: it chooses more by the form of the text, such as the syntax of code.

Leads to

  • DeepSeek-V3 (December 26, 2024)

    Inherited, with no proven direct link

    Both are open sparse mixture-of-experts models that activate only part of their parameters for each token

Sources

Original source

How to cite

“Mixtral 8x7B.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/mixtral

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline