Hugging Face Releases "The Ultimate MoE Training Handbook": What Is Inside the Free 70,000-Word Guide

Hugging Face has released a free, open guide to a skill that most AI teams keep in a few heads: how to train a Mixture-of-Experts (MoE) model. "The Ultimate MoE Training Handbook" went live on November 5, 2026 as a Hugging Face Space. Its authors are Nouamane Tazi, Leandro von Werra and Thomas Wolf, and its text says it is a hands-on guide to training MoE models with Megatron-LM. Wolf, a Hugging Face co-founder, announced it with a clear pitch: his team ran about 800,000 GPU hours of experiments and wrote everything down.

This post explains what the handbook contains, which of its numbers stand out, and where a reader should start. We read the handbook's published text, not just the announcement. We did not rerun its experiments, and the figures below are the authors' reports.

TL;DR: the handbook at a glance

Why a training handbook, and why now?

The opening of the handbook makes a market claim. On the Artificial Analysis index, it says, 18 of the 20 highest-scoring open-weights models are mixtures of experts. The two dense exceptions it names are Nanbeige4.3-33B and Qwen3.8 27B. Wolf puts it more bluntly in his post: "Dense is basically over." If that is right, then knowing how to train a sparse model is no longer a niche skill.

That matches what we have seen in our own coverage. Recent open releases such as the DeepSeek V4 Flash vision model, the Baidu ERNIE 5.5 Flash Base and the LongCat 2.5 agent model are all sparse designs. A reader who wants to know what "14B active" or "top-8 of 256 experts" means can use a plain explanation, and our dictionary entry on Mixture of Experts gives the short version: a router sends each token to a few expert sub-networks, so a model can hold many parameters but use only a few per token.

The handbook goes much further than a definition. Wolf says the know-how for a good MoE "still lives mostly in the heads of a handful of teams." The handbook tries to move it into the open, and it is organised around the questions that training teams hit.

What questions does the handbook answer?

Wolf lists seven questions in his announcement. They are a good map of the document:

1. How many experts should a model have, how big should they be, and should some be shared?
2. Why is half of a training step spent in all-to-all communication, and how can it be hidden?
3. Why is a run 25 times slower with no error in the logs?
4. Which router function should be used (softmax, sigmoid or sqrt(softplus)), and why keep it in fp32?
5. Should balancing use an auxiliary loss or DeepSeek's loss-free bias?
6. Should a team upcycle a dense checkpoint or train from scratch?
7. Can a model be served with fewer experts per token than it was trained with?

The published table of contents is structured as a path from first run to production. Chapter 1 is the introduction. Chapter 2 covers setup and a first run, from hardware tests to a dense baseline and a first MoE. Chapter 3 is the anatomy of MoE training, with sections on monitoring, router scoring, routing strategy and load balancing. Chapter 4 moves from training to serving, and a short conclusion collects the recommended flags. An appendix follows. We read the chapters through the serving and conclusion sections. The sections of chapter 3 after load balancing, including the systems material, we read only in part.

How does the setup chapter start?

The handbook does not begin with theory. It begins with a cluster check, and that is useful advice for any team.

Its reference environment is AWS P5 nodes, each with eight H100 80GB GPUs, connected by EFA. It pins software versions, including NGC PyTorch 26.06, EFA installer 1.49.0, aws-ofi-nccl v1.19.2 and a specific Megatron commit. It tells readers to run dcgmi diag -r 1 on every node before training. In one example, a node tested at a deeper level showed thermal violations and PCIe and memory failures on three GPUs.

It also warns against trusting one network test. In its example, a slow allocation measured 39.84 GB/s on all-to-all but 398.52 GB/s on all-reduce, so an all-reduce check alone would have passed a bad allocation. The handbook recommends recording an all-to-all baseline for each node count, and it lists baselines of 77.14 GB/s on two nodes, 39.25 on four and 30.91 on eight. It suggests requiring about 90% of baseline plus a correctness check before a run starts.

The first model is small on purpose. The dense baseline has 16 layers and about 1.22B parameters. The first MoE has 64 experts, top-8 routing and an expert width of 1,024, which makes eight active experts match the dense feed-forward size. Its totals are 6.86B parameters in all and 1.221B active. A parity check shows the idea behind a good baseline: a one-expert upcycled MoE stayed within 0.00087 of the dense run over 15 steps, while a non-upcycled initialisation differed by 2.72 at iteration 4.

Which router settings does it recommend?

Section 3.1 is the clearest example of the handbook's style. It gives a recommendation and the evidence behind it.

• Router precision. Use fp32 for the router, with Megatron's --moe-router-dtype fp32. The handbook notes that a bf16 router diverged in the Switch Transformers work. If router logits drift upward, it points to a z-loss, and it cites ST-MoE's 1e-3.
• Renormalisation. Keep it on when each token picks more than one expert. The handbook cites a result where it lowered validation loss from 2.219 to 2.183 at a granularity of 8. Top-1 softmax needs a pre-softmax setting or the router gets no gradient.
• Scoring function. Softmax, sigmoid and sqrt(softplus) keep the same top-k ranking but change the weights. Sigmoid or sqrt(softplus) is needed to use an expert bias, because Megatron rejects the bias with softmax.
• Routed scaling. For equal weights on uncorrelated experts, roughly the square root of k restores output scale, but shipped models disagree. The handbook lists DeepSeek-V3 at 2.5 with k of 8 and Kimi K3 at 1.0 with k of 16.

The routing-strategy section compares token choice, expert choice and hash layers. Expert choice gives balanced loads but, the handbook says, leaks future-token information. In a 575M test it reached about 2.844 loss against about 2.89 for token choice. Hash routing, which fixes a token-to-expert table, helped only with a single MoE layer in the work it cites, and the handbook suggests it mainly to reproduce DeepSeek-V4's first layers.

Aux loss or loss-free bias: what do the numbers say?

Diagram of a router sending tokens to expert blocks, illustrating MoE training load balancing between experts

This is the question most teams argue about, and the handbook gives numbers instead of a verdict.

Imbalance has a direct speed cost. In the handbook's runs, forced-even routing reached 149.4 TFLOP/s and forced-skewed routing fell to 100.5. The learned router started near the worst case and reached 125 TFLOP/s after 120 iterations.

For the balancing method itself, it cites Wang et al. At a coefficient of 1e-3, the auxiliary loss gave a MaxVio of 0.72 and a perplexity of 9.56. The bias method gave a MaxVio of 0.04 and a perplexity of 9.50. With no balancing, MaxVio rose to about 5.1. It also cites a study where test loss rose as the aux coefficient went from 1e-3 to 0.1.

But the handbook does not declare the bias the winner. It cites Li et al., who found that at 1,024 experts a 1e-3 bias update rate let the busiest expert reach about 50 times the mean load, while an aux loss of 1e-2 kept it under about 7 times. Its practical advice is to start with aux_loss at 1e-3, watch MaxVio and language-model loss, and adjust. It also advises running a forced-even control once to set a speed target, and never training with it.

How many experts, and should some be shared?

The handbook reports a granularity result from an 18-layer model with fixed parameters and compute: loss fell from 2.334 to 2.297 at 128 experts of one quarter width with top-8, and finer splits gave no further gain. It describes a "latent MoE" design that, per NVIDIA's 73B-A8B ablation as cited, scored 4.6 points higher on MMLU-Pro at equal expert parameters and traffic.

Shared experts get a mixed verdict. Loss improved by about 0.01 to 0.02 in some from-scratch models, while OLMoE and another sweep found no effect or a slightly worse one, and Mellum2 found no quality gain and slower inference. The handbook also gives a way to see the cost: with 32 experts and 4 active, reserving one shared slot leaves 4,495 combinations for the rest, against 35,960 for four picked from all 32.

Why was one run 25 times slower with no error?

Network fabric of connected nodes, showing MoE all-to-all communication and a fallback that slowed a run

This is the story engineers will remember. The handbook says two runs set FI_PROVIDER=efa and named an OFI plugin, yet NCCL still logged "Using network Socket." Traffic had fallen back to plain TCP, and those runs were about 25 times slower than an expert-parallel run on EFA. Nothing in the error logs said so.

The fix it describes is specific. Setting the fabric variables is not enough. NCCL reached EFA only after the host's RDMA libraries came first on the library path, and naming the plugin by path or by name still left it on sockets. The check is to run with NCCL_DEBUG=INFO and read the startup log for the transport NCCL picked. For a related look at how communication libraries shape RL post-training, see our post on NVIDIA's Reshard collective.

The handbook also covers how to hide all-to-all time. With --overlap-moe-expert-parallel-comm, one micro-batch's forward pass runs beside another's backward pass, and adding --delay-wgrad-compute lets the backward dispatch wait only on input gradients. On two nodes it reports throughput up to 35% higher, with rank 0 reserving 5 to 10 GB more memory. Rewiring the block can also help: it cites ScMoE training 1.12 to 1.14 times faster, and LongCat-Flash's exposed all-to-all share falling from 25.3% to 8.4%.

Upcycle a dense model or train from scratch?

The handbook gives two data points, and they differ. In OLMoE, a run from scratch caught up with the upcycled MoE after 500B tokens, which is 25% of the dense model's 2T training tokens. In the sparse-upcycling work of Komatsuzaki et al., catching up took 120% of the dense checkpoint's compute. The upcycled run starts ahead on all four metrics in OLMoE's logs. The practical reading: upcycling buys an early lead, and how long it lasts depends on the budget and recipe.

Can you serve with fewer experts than you trained with?

Router junction diverting a token to one of several paths, illustrating MoE serving with fewer experts per token

Chapter 4 turns to inference, and the results are the most reusable for people who run open models rather than train them.

The test model is Qwen3.6-35B-A3B, with 256 routed experts trained at top-8. Against a trained baseline of 81.65% MMLU at top-8:

Top-4 halves the routed parameters read per token, from 1.006B to 0.503B. But the handbook shows the result depends on the model. On Gemma-4-26B-A4B, where the top 8 of 128 experts hold 0.302 of the mass, dropping renormalisation at top-6 raised perplexity by 0.7%, against 356% on the Qwen model. On OLMoE-1B-7B, serving top-16 instead of the trained top-8 moved a nine-benchmark average from 41.35 to 41.09, within the reported error bars. The authors also note that metrics can disagree. For local-model readers, our Gemma 4 26B-A4B MLX speed-up post shows the same active-parameter idea from the inference side.

The chapter adds other numbers. It says DeepSeek-V4.1-Flash uses expert parallelism of 2 to 8 for prefill and 64 to 128 for decode. At 64 requests per GPU, it reports about 15,700 tokens per second per GPU at EP64 against about 12,600 at EP8, and the two nearly meet at 256 requests per GPU. It says gpt-oss-20b drops from 39.0 GiB to 12.8 GiB when only the routed experts are stored in 4 bits. Our DeepSeek V4.1 Flash KV-cache post covers another side of that model's memory story.

What does it say about reinforcement learning?

The handbook flags a risk that matters for anyone doing RL on MoE models. The rollout engine and the trainer can route the same token to different experts. It cites Ma et al., who found about 10% of routers chose different experts and 94% of tokens differed in at least one layer. Another study found about 10% of activated experts changed after one update. A routing-replay method cut train-inference KL from 1.5e-3 to 7.5e-4, with under 3% added rollout latency, but the handbook says reward gains from replay are contested. Readers choosing a stack can compare it with our overview of open-source RL-as-a-service stacks.

What are the limits of the handbook?

The handbook is honest about its own limits, and so should a summary be.

• It is long. Wolf himself says to budget a few days. The page lists a reading time of one to two days.
• It leans on single ablations. The conclusion notes that many recommendations rest on single experiments and that much is still open. Its advice is to add one group of flags at a time and measure each, because gains may not add up.
• It is tied to one stack. Its commands are Megatron-LM flags and its hardware is H100s on AWS with EFA. Teams on other frameworks must translate.
• Many numbers come from other papers. Where the handbook cites outside studies, such as the balancing and upcycling results, those are the cited authors' findings and the handbook's summary of them.
• We did not reproduce anything. Our read covered the text, not the code or interactive figures.

Who should read what first?

• Engineers about to train an MoE: start with chapter 2 to build the hardware and baseline checks, then the router and load-balancing sections.
• Platform and infrastructure teams: read the all-to-all benchmark advice and the 25x slowdown story, then the overlap section.
• People who serve open models: go straight to chapter 4 and the experts-per-token tests.
• Learners: read the introduction and the router sections for the clearest explanation of why sparse models work, then browse the interactive figures.

If you want guided practice rather than a self-paced read, see our workshops. For how this sits in the wider open-model landscape, our Qwen4 architecture preview shows where sparse designs are heading.

What this means for builders

For most readers the value is not a new model. It is a map. When a lab says a model has 256 experts at top-8, the handbook shows what trade-offs sit behind those numbers, which flags to measure, and which failures hide quietly in logs. A free reference with this much measured detail lowers the entry cost for teams that could not afford to rediscover it. It does not remove the compute cost: Wolf's own number is 800,000 GPU hours for the team's experiments alone.

Related reading

• DeepSeek V4 Flash vision: 305B open model
• DeepSeek V4.1 Flash and the KV-cache memory reduction
• Baidu ERNIE 5.5 Flash Base: 280B open-weights MoE
• LongCat 2.5: 1.6 trillion parameter agent model
• Gemma 4 26B-A4B on Mac with MLX
• NVIDIA Reshard collective for RL weight refit
• Open-source RL-as-a-service stacks
• Primary sources: the handbook on Hugging Face and Thomas Wolf's announcement on X

Details reflect the handbook text and announcement as of November 7, 2026. The handbook may be revised after publication.