Mixtral of Experts

Mixtral starts from the same Transformer decoder used in Mistral 7B — with grouped query attention, sliding window attention, and RoPE positional encodings — but makes one decisive change: every feedforward (FFN) sub-block is replaced by a Mixture-of-Experts (MoE) layer.

In a standard Transformer, the FFN block is a single large network that every token must pass through. Think of it as a single highway lane: no matter the vehicle, everyone merges into the same lane. In Mixtral, that single lane is replaced by 8 parallel lanes (experts), and a traffic controller (the router) directs each vehicle (token) to the 2 lanes that best match its needs.

Each expert is a SwiGLU feedforward network — the same architecture used inside Mistral 7B — with its own independent set of weights. The key insight is that although the model has 8 experts per layer (totaling ~47B parameters), any individual token only activates 2 experts (~13B active parameters). This is why the approach is called sparse: most of the model is dormant for any given token.

The router (also called the gating network) is the brain of the MoE layer. For each token, it must answer one question: "Which 2 of these 8 experts should handle this token?"

Mechanically, the router is a simple linear layer followed by a Top-K selection and softmax. The input token's hidden state $x$ is multiplied by a weight matrix $W_g$ to produce 8 scores (one per expert). The Top-2 scores are kept; all others are set to $-\infty$. A softmax over the surviving scores produces the gating weights — these weights determine both which experts are activated and how much each expert's output contributes to the final result.

The beauty of this design is its simplicity. The router has no memory, no recurrence, no special tricks — just a single matrix multiply and a selection step. Yet through training, it learns to route tokens to appropriate experts based on their content.

The central claim of Mixtral is that a sparse model with 47B total parameters and only 13B active parameters per token can match or beat a dense model with 70B parameters that are all active for every token.

This distinction between sparse parameter count (total) and active parameter count (per token) is fundamental. Think of it as the difference between a library's total collection and the books you actually take off the shelf for a given research question. The library is vast, but your reading cost is small.

In practice, Mixtral's inference compute is proportional to its active parameters (13B), not its total parameters (47B). This means it processes tokens at roughly the speed of a 13B dense model, while drawing on the knowledge capacity of a much larger one. The trade-off is memory: the full 47B parameters must be stored, though this is still smaller than Llama 2 70B.

Mixtral was evaluated across a comprehensive set of benchmarks covering commonsense reasoning, world knowledge, reading comprehension, mathematics, code generation, and multilingual understanding.

The headline result is striking: with only 13B active parameters, Mixtral matches or outperforms Llama 2 70B — a model that uses 5× more active parameters — on almost every benchmark. On mathematics (GSM-8K: 74.4% vs 69.6%) and code generation (HumanEval: 40.2% vs 29.3%; MBPP: 60.7% vs 49.8%), the gap is enormous.

Compared to GPT-3.5, Mixtral performs similarly on MMLU (70.6% vs 70.0%) and outperforms it on code (MBPP: 60.7% vs 52.2%). The instruction-tuned variant, Mixtral Instruct, achieves an MT-Bench score of 8.30 — surpassing GPT-3.5 Turbo (8.32 is comparable), Claude-2.1, and Gemini Pro on human evaluation via the LMSys Chatbot Arena.

Mixtral was pretrained with a significantly increased proportion of multilingual data compared to Mistral 7B. The extra capacity provided by the 8 experts allows the model to absorb multilingual knowledge without sacrificing English performance.

The results on French, German, Spanish, and Italian benchmarks are particularly impressive. On MMLU in French, Mixtral scores 70.9% compared to Llama 2 70B's 64.3%. On German MMLU, it achieves 71.5% versus 64.2%. The pattern repeats across all four languages and all three benchmark types (ARC-Challenge, HellaSwag, MMLU), with Mixtral consistently outperforming a model that uses 5× its active compute.

This suggests that the additional expert capacity provides "room" for the model to store language-specific knowledge in specialized experts, while shared linguistic structure is captured by experts activated across languages.

A natural question is whether experts specialize by domain — does one expert handle mathematics while another handles biology? The authors investigated this by measuring expert selection distributions across different subsets of The Pile validation dataset: ArXiv, DM Mathematics, GitHub, Gutenberg, PhilPapers, PubMed Abstracts, StackExchange, and Wikipedia.

The surprising answer is no — there is no clear domain specialization. The distribution of expert assignments looks remarkably similar across topics like ArXiv (LaTeX papers), PubMed (biology), and PhilPapers (philosophy). Only DM Mathematics shows a slightly different pattern, likely because it is a synthetic dataset with limited natural language diversity.

Instead of domain specialization, the router exhibits syntactic behavior. When visualizing token-level routing on code, mathematics, and English text, the patterns follow syntax rather than semantics: Python's self keyword consistently routes to the same expert, indentation tokens cluster together, and consecutive tokens within the same syntactic unit often share experts. This "positional locality" is especially pronounced in the middle and final layers of the model.

The base Mixtral model is a next-token predictor. To turn it into a helpful assistant, the authors applied a two-stage fine-tuning recipe: supervised fine-tuning (SFT) on an instruction dataset, followed by Direct Preference Optimization (DPO) on a paired feedback dataset.

DPO is a simpler alternative to RLHF: instead of training a separate reward model and running reinforcement learning, DPO directly optimizes the language model on pairs of (preferred response, rejected response). The result is a model aligned with human preferences without the complexity of PPO.

Mixtral Instruct achieved an MT-Bench score of 8.30 and reached an Elo rating of 1121 on the LMSys Chatbot Arena (as of December 2023) — placing it above Claude-2.1, every version of GPT-3.5 Turbo, Gemini Pro, and Llama 2 70B Chat. It was the highest-ranked open-weight model at the time.

The authors also measured biases using BBQ and BOLD benchmarks. Mixtral shows higher accuracy on BBQ (56.0% vs 51.5% for Llama 2 70B), indicating fewer social biases, and displays more positive sentiment with similar variance on BOLD.

Mixtral was trained with a context window of 32,768 tokens and supports fully dense attention across the entire window. To test whether the model can actually use all of this context, the authors ran the passkey retrieval task: a random passkey is inserted at a random position in a long prompt, and the model must retrieve it.

Mixtral achieves 100% retrieval accuracy regardless of where the passkey is placed or how long the prompt is. This confirms that the model does not suffer from the "lost in the middle" problem that affects some long-context models.

Additionally, perplexity on the proof-pile dataset decreases monotonically as context length increases — evidence that the model genuinely leverages longer context rather than ignoring distant tokens.

Mixtral's contribution is not any single technique — MoE existed, SwiGLU existed, top-K routing existed. The contribution is the engineering proof that these ideas, combined carefully in an open-weight model, produce a system that reshapes the cost-performance frontier. It showed the community that sparse models are not just a research curiosity but a practical path to deploying powerful language models at lower cost.

The line from Mixtral to DeepSeek-V3 — which scaled MoE further — demonstrates that this was not a dead end but the beginning of a new architectural paradigm for frontier LLMs.