Every few years a single architectural idea quietly becomes the default. For frontier language models, that idea is the Mixture of Experts, or MoE. Mixtral, DeepSeek-V3, Qwen3’s large variants, Llama 4 and Grok-1 are all publicly documented MoE designs, and GPT-4 is widely reported to be one as well. If you follow model releases, you will see the same pattern: a huge “total parameters” number next to a much smaller “active parameters” number.
That gap is the whole story. MoE lets a model hold far more knowledge than it ever uses on a single token. This article explains how that works, what the router actually does, why the economics favor it, and what leaders should ask when evaluating MoE-based models.
The Core Idea: Capacity Without Proportional Compute
In a standard (dense) Transformer, every token passes through every parameter in every layer. Doubling parameters roughly doubles the compute for every token, in training and in inference.
MoE breaks that coupling. In a Transformer block, the feed-forward network (FFN) holds the majority of the parameters. MoE replaces that single FFN with many smaller FFNs, called experts, plus a small router (or gating network). For each token, the router picks a few experts, and only those run. The attention layers are usually shared and stay dense.
A useful analogy is a hospital. A dense model is a single doctor who must know everything and sees every patient with the full weight of that knowledge. An MoE model is a hospital with many specialists and a triage desk. Total expertise in the building is enormous, but each patient only consumes the time of the two or three specialists they need.
How Routing Works
At its core, routing is an automated matchmaker: it takes a single token representation and decides which small subset of experts should handle it.
Technically, the router is almost surprisingly lightweight, often just a single linear projection matrix. Whenever a token arrives at an MoE layer, the model runs a straightforward four-step cycle:
Score (Evaluate affinity): The token’s incoming vector is multiplied by the router’s weights, generating a raw compatibility score (a logit) for every available expert. A softmax function then converts these logits into normalized confidence scores between 0 and 1.
Select (Pick the top-$k$): The router picks the $k$ highest-scoring experts and discards the rest. Different architectures tune this parameter according to their design goals:
$k = 1$ (Switch Transformer): Aggressively minimizes compute by sending each token to exactly one expert.
$k = 2$ (Mixtral 8x7B, GShard): Uses a primary and a secondary expert to strike a balance between parameter capacity and compute.
Fine-grained selection (DeepSeek-V3): Breaks the network into many smaller pieces, routing to 8 out of 256 specialized experts, alongside one always-active shared expert.
Compute (Process in parallel): Only the selected experts perform forward-pass calculations on the token. The remaining 90%+ of the layer’s parameters sit idle for that specific step, saving arithmetic work.
Combine (Weighted merge): Each active expert returns its transformed output. The router scales each output by its normalized score from Step 1, sums them together, and passes the blended vector to the next layer.
This cycle runs independently for every token at every MoE layer. A word like “contract” might route to Experts 3 and 7 in layer 8 to parse syntax, but trigger Experts 14 and 42 in layer 24 to handle semantic context.
Routing happens per token and per layer. The same sentence can send “invoice” to one set of experts at layer 10 and a different set at layer 20. Experts are not hand-labelled departments such as “law” or “medicine.” Studies of trained MoE models show specialization that is often syntactic or token-pattern based rather than a clean topical split. Treat the specialist analogy as intuition, not literal fact.
Sparse Activation: Total vs. Active Parameters
Sparse activation means only a fraction of the model runs for each token. This is why two numbers matter:
| Model | Total parameters | Active per token |
|---|---|---|
| Mixtral 8x7B | about 47B | about 13B |
| DeepSeek-V3 | 671B | 37B |
| Llama 4 Maverick | about 400B | 17B |
The compute cost per token (FLOPs) tracks the active parameters, while the model’s capacity to store knowledge tracks the total. A model with 671B total and 37B active spends roughly the arithmetic of a 37B dense model per token while drawing on a far larger pool of learned parameters.
One caveat is critical for anyone planning infrastructure: memory does not shrink. All experts must be resident in accelerator memory, or fetched quickly, because any token might need any expert. MoE reduces compute per token, not the memory footprint of the weights.
Why Frontier Models Use MoE
1. Better quality for the same training compute. Research including the Switch Transformer and GLaM papers showed that sparse models reach a given quality with substantially less training compute than dense counterparts. Scaling laws, covered earlier in this series, tell us that more parameters help. MoE makes extra parameters cheap in FLOPs.
2. Lower inference cost per token. Serving cost is dominated by compute and memory bandwidth per generated token. Activating 5 to 10 percent of the weights cuts compute sharply, which directly improves cost per million tokens and latency at scale.
3. A path to continued scaling. Dense models hit a wall where training and serving cost grows linearly with size. Sparse models keep growing total capacity while holding per-token cost nearly flat, which is why the largest open and proprietary models have converged on the pattern.
4. Natural parallelism. Experts can be placed on different devices (expert parallelism), spreading the weights across a cluster in a way that matches how the model is structured.
The Hard Parts: What MoE Costs You
MoE is not free. Leaders should know the trade-offs behind the headline numbers.
Load balancing. A naive router tends to collapse: a few experts get most tokens and the rest never train. Training adds an auxiliary load-balancing loss to encourage even usage, or, as in DeepSeek-V3, an auxiliary-loss-free bias adjustment that nudges the router without degrading quality. Each expert also has a capacity limit, and tokens that overflow can be dropped or rerouted.
Communication overhead. When experts live on different GPUs, tokens must be dispatched to them and results gathered back, an all-to-all communication pattern. On weak interconnects this becomes the bottleneck. Cluster network design matters as much as chip count.
Training instability. The router’s discrete choices can make training less stable. Techniques such as router z-loss, careful initialization and higher-precision routing computations are used to tame it.
Serving complexity. Small-batch inference touches many experts, so memory bandwidth is poorly used. MoE shines at high-throughput serving with large batches and is less efficient for single-user, low-batch deployments, especially on a single GPU.
Fine-tuning. Early MoE models overfit more easily during fine-tuning, and tuning routing behavior needs care. Parameter-efficient methods and more recent recipes have improved this, but it remains a consideration.
Design Variations Worth Knowing
- Fine-grained experts: many smaller experts allow more flexible combinations per token. DeepSeekMoE popularized this approach.
- Shared experts: one or more experts always run, capturing common knowledge so routed experts can specialize.
- Expert-choice routing: instead of tokens choosing experts, experts choose tokens, which balances load by construction.
- Dense and MoE layers interleaved: some models alternate dense and MoE layers to balance stability and efficiency.
What This Means for Decision-Makers
Read model cards correctly. “671B parameters” does not mean 671B-class cost per token, and “37B active” does not mean it fits on a 37B-class machine. Ask for both numbers, and ask about memory requirements.
Budget for memory and networking, not just FLOPs. Self-hosting an MoE model means provisioning for total parameters and fast interconnects. For many organizations, a managed API remains the more economical route.
Match the deployment to the architecture. High-volume, batched workloads benefit most from MoE. Latency-sensitive, low-volume or edge workloads may be better served by a smaller dense model.
Compare on outcomes. Active parameters are a cost proxy, not a quality guarantee. Evaluate on your own tasks, measuring quality, latency and cost per request together.
Expect more of it. Open-weight MoE releases make frontier-class capability more accessible, but the serving expertise required is a real differentiator. Plan for skills in inference optimization, or choose partners who have them.
Key Takeaways
- MoE replaces a Transformer’s dense feed-forward layer with many experts and a router that selects a few per token.
- Sparse activation means compute scales with active parameters while knowledge capacity scales with total parameters.
- Memory requirements still follow total parameters, so MoE saves compute, not memory.
- The main engineering challenges are load balancing, communication, stability and serving efficiency.
- Frontier models adopt MoE because it delivers more quality per training dollar and lower cost per served token.
Conclusion
Mixture of Experts is a straightforward bargain: spend memory and engineering complexity to buy capacity at a fraction of the per-token compute. As scaling pressures grow, that bargain has become the standard for frontier models. Understanding routing and sparse activation lets leaders read model announcements critically, plan infrastructure realistically, and choose the right architecture for each workload.
Figures for specific models are drawn from publicly released papers and model cards and are approximate; GPT-4’s architecture has not been officially confirmed by OpenAI.
