Amazon HOPE: Second-Order MoE Expert Pruning With Open awslabs Code

Amazon published HOPE (Higher-Order Pruning of Experts) on November 9, 2026: a second-order way to delete idle experts from mixture-of-experts (MoE) language models while preserving clusters that routinely co-activate. The team ships a three-step CLI and Python API on GitHub as awslabs/HOPE, with the method explained on Amazon's Hugging Face blog post and in arXiv:2609.18916. At 50% pruning, HOPE reported up to +6.1% on agentic coding tasks versus the next-best first-order baseline and, on one Qwen3.5-35B run, a 42.3 SWE-Bench Pro score that nearly matched the unpruned 41.8.

If you run large MoE models locally or on a fleet, most VRAM is expert MLP weights you never touch on a given token. HOPE is aimed exactly at that waste line — without asking you to retrain or rewrite inference servers.

Why first-order expert pruning breaks cooperating experts

MoE layers route each token to a small active set (often top-8) out of dozens or hundreds of experts per layer. Amazon's example: Qwen3.5-122B keeps 256 experts in each of 48 MoE layers, and about 95% of the model's on-disk parameter mass sits in those expert MLPs even when only eight fire per token.

Expert pruning permanently deletes experts to shrink checkpoints and VRAM. That is attractive because it needs no fine-tuning, avoids dynamic expert load/unload latency, and stacks with quantization — themes we cover in semantic caching for LLM apps and vLLM versus SGLang when you are squeezing cost per request.

The catch is which experts to cut. First-order methods (REAP, MAN, EAN, frequency heuristics) score experts independently on a calibration set and drop the lowest ranks per layer. That ignores a stable empirical fact Amazon highlights: experts form cooperative groups that get selected together, a pattern other MoE papers also observe early in training.

HOPE MoE expert pruning compares first-order scores with pairwise expert cooperation Image: Amazon Science (Hugging Face blog)

Removing expert A can look safe in isolation while breaking expert B that relied on A whenever both routed. At aggressive prune rates, those broken links compound. HOPE treats that as a quadratic optimization problem instead of N independent rankings.

For MoE basics and routing math, pair this with our KV cache memory explainer — MoE shrinks static weights, while KV cache still scales with context at runtime.

How HOPE builds the F-matrix and picks a prune set

HOPE's derivation (Appendix A in the paper) starts from pruning error: deleting experts changes which backup experts receive traffic, which makes exact minimization intractable. Amazon introduces a surrogate Z that upper-bounds the dominant error term and expands into pairwise products of gate weights and expert output norms.

Calibration is one forward pass per layer, recording which experts activate, gate weights, output norms, and co-activation counts. Those statistics assemble an F-matrix where:

• Diagonal entries approximate solo importance (and nearly recover REAP's ranking).
• Off-diagonal entries score cooperative contribution when two experts share a token's active set.

HOPE then solves a binary quadratic program per layer: pick exactly P experts to delete minimizing p^T F p, using a continuous relaxation and rounding. Solving takes on the order of one to two seconds per layer — negligible versus calibration, per Amazon's post.

GLM-4.5-Air HOPE interaction matrix showing cooperating MoE expert clusters Image: Amazon Science (Hugging Face blog)

The blog visualizes GLM-4.5-Air layers where bright blocks along clustered axes show tight cooperation bands. That picture is why HOPE sometimes keeps a slightly lower solo-scored expert: it protects a cluster first-order cuts would dismantle.

Limitations Amazon states plainly: HOPE is layer-local (cross-layer HOPE is preliminary in the paper), focuses on pairwise not triplet interactions, and its lead over REAP shrinks below ~40% pruning. For giant frontier MoE stacks like Tencent HY4, expert pruning is complementary to architectural choices (latent attention, grouped routing) already baked into the base model.

Benchmarks: where HOPE wins and ties the unpruned model

Amazon tested Qwen3.5-122B, Qwen3.5-35B, and GLM-4.5-Air with calibration from Evol-CodeAlpaca and SWE-Bench Verified trajectories, then evaluated on Tulu3 suites plus SWE-Bench Pro agentic coding.

HOPE MoE pruning benchmark ranks versus REAP at aggressive 50 percent expert cuts Image: Amazon Science (Hugging Face blog)

Headline numbers from the post:

SWE-Bench Pro matters for practitioners because it stresses multi-turn tool use — the regime where diverse expert combinations show up. That lines up with Amazon's story: agentic workloads benefit most when cooperative structure survives pruning.

Amazon also notes an occasional pruned-beats-full effect (noise experts removed), not unique to HOPE. Code-focused calibration did not catastrophically harm non-code Tulu3 tasks in their runs — useful if you plan to specialize a pruned checkpoint for coding agents without maintaining two full-size MoE copies.

Compare this compression axis to kernel-side work like SGLang Muse mega-kernels: HOPE shrinks which experts exist; Mega kernels speed how matmuls run on whatever weights remain.

How to run HOPE on your own MoE checkpoint

Amazon documents a CLI in three commands (also exposed as Python helpers):

hope calibrate \
    --model-path Qwen/Qwen3.5-35B-A3B \
    --prompts calibration_prompts.txt \
    --out-path observations.h5

hope solve \
    --obs-path observations.h5 \
    --prune-frac 0.50 \
    --out-path pruneset.json

hope prune \
    --model-path Qwen/Qwen3.5-35B-A3B \
    --pruneset-path pruneset.json \
    --out-path ./Qwen3.5-35B-A3B-pruned-50

Calibration prompts should resemble production traffic if you care about a niche (support bots versus codegen). The HDF5 stores the F-matrix; solve emits pruneset.json; prune writes a standard Hugging Face folder you can point vLLM or SGLang at — no custom expert-loader patches per Amazon.

Practical checklist before you run it on a shared GPU box:

1. Disk and RAM to hold the dense MoE source once.
2. A calibration file long enough to cover routing modes you care about (Amazon used code-heavy sets for agentic gains).
3. A prune fraction decision: 50% is where HOPE's memory win is largest but also where routing distortions hurt if calibration lied about your traffic.

If you are already experimenting with StepFun P2 distillation or other Qwen-family tweaks, treat HOPE as an orthogonal lever — it does not change tokenizer or attention code, only which expert weights exist.

Hybrid-architecture follow-ups live in a separate Amazon repository linked from the blog for readers pruning next-gen MoE hybrids; check that README if your model mixes dense and sparse blocks.

What this means for builders and operators

HOPE does not replace quantization, speculative decoding (our guide), or KV compression. It attacks a different bill item: the enormous expert parameter tensor that makes MoE attractive at quality but painful at deploy time.

Operators running Qwen3.5 or GLM-4.5 class models should read HOPE as a cheap experiment: a day of calibration and solve passes can tell you whether you can halve expert VRAM on agentic workloads before you commit to buying more GPUs. Researchers comparing pruning papers now have an open second-order baseline with reproducible CLI steps — rare in a space full of leaderboard-only claims.

Watch the usual caveats: pruning is irreversible without keeping the original checkpoint; calibration mismatch still kills quality; and layer-local pruning cannot fix a router that was broken before you started.

Choosing a calibration set without fooling yourself

Amazon's README pattern uses a plain-text calibration_prompts.txt. Quality of that file is the hidden hyperparameter:

• Match your traffic shape. If production is 80% single-turn FAQ and 20% multi-tool coding, a file that is 100% SWE trajectories will over-prune experts that matter on short turns.
• Length mix matters. Expert routing differs on 500-token prompts versus 32K RAG dumps. Include both if both exist in prod.
• Re-run solve when the model updates. Router statistics drift across checkpoint revisions even within Qwen3.5 patch releases.

HOPE's diagonal almost equals REAP (Spearman 0.988 in the blog). When HOPE beats REAP, the win comes from off-diagonal terms — cooperators REAP would split. If your calibration never activates those pairs, HOPE collapses toward REAP behavior and you paid QP time for little gain.

For deployment math: at 50% pruning on a 122B-class MoE, you are not saving 50% of end-to-end latency — you shrink static weights and improve batch size headroom. Pair HOPE with FP8 weights and PagedAttention tuning rather than expecting a magic throughput doubling.

REAP, MAN, and when first-order methods are enough

Amazon benchmarks against REAP, MAN, EAN, and frequency cuts. At 10–20% pruning HOPE's average rank advantage shrinks; first-order cuts are simpler (no QP) and may be fine when you only need a modest VRAM shave.

Use HOPE when:

1. You are trying to fit a model that already OOMs at your context target.
2. Your eval includes agentic multi-step workloads (tool calls, repo edits).
3. You can afford one offline calibration job per model revision.

Stick with REAP or quant-only when you need a quick A/B on a stable chat model with short prompts.

Governance and licensing notes

HOPE ships under Apache 2.0 on awslabs/HOPE per the GitHub link in Amazon's post. That is distinct from model licenses on Qwen or GLM weights — pruning does not change their terms, but redistributing pruned checkpoints still inherits the base model's license.

Inside enterprises, treat pruned artifacts like any other derived model: log which pruneset.json produced which checkpoint, store calibration prompts for audit, and keep an unpruned rollback copy until production metrics clear.

Related reading

• What Is the KV Cache in LLM Inference?
• vLLM vs SGLang: Inference Server Comparison
• Semantic Caching for LLM Apps
• Agent Sandboxes for Coding Agents Compared
• Tencent HY4 Official Release