Half the Experts, Full Power? What Pruned MoE Models Lose in German

What this is about

Large language models built on the Mixture of Experts (MoE) principle are fast, but they need a lot of memory. They're made up of hundreds of specialized subnetworks, the "experts." Only a few of them are active for any given word, but all of them have to sit in memory. One obvious trick is expert pruning: you remove the experts that supposedly barely get used, and you don't retrain the model afterwards. Hugging Face is now full of pruned variants carrying the REAP label. Amazon has just unveiled HOPE, a method that it says even scores slightly higher than the original on a coding benchmark with half the experts removed.

Our argument: "lossless" only applies to the tasks a model was pruned and tested on, and that's almost always English code. Nobody has published measurements of what pruning does to German. But existing research gives some clues about where the risks are.

How an MoE model picks its experts

Every MoE layer has a router, a kind of switch. For each token, meaning each word or word fragment, it decides which experts should do the computing and weights their contributions. According to the HOPE paper, Qwen3.5-122B-A10B has 48 MoE layers with 256 experts each, 8 of which are active per token. "A10B" stands for roughly 10 billion active parameters per token.

Nearly all the memory goes to the experts. Amazon says 116 of the 122 billion parameters are expert weights. That's about 95 percent of the roughly 250 GB the model needs at full precision. Halve the number of experts, the paper says, and you need about 125 GB. Our back-of-the-envelope math for 4-bit quantization (where the numbers are stored at lower precision): instead of roughly 65 to 70 GB, you'd need more like 35 GB. That math is exactly why the approach appeals to people who want to run large models on their own hardware at today's RAM prices.

REAP: rarely contribute much, and you're out

REAP (Router-weighted Expert Activation Pruning) comes from Cerebras and was accepted at ICLR 2026. It gives each expert a score. To get it, you run calibration data, a sample of typical inputs, through the model. For every token an expert was selected for, REAP measures how heavily the router weighted it and how large its contribution was. The average of those values is the score, and the lowest-scoring experts get deleted. Because the average only covers tokens where the expert was actually active, rare but important specialists are supposed to survive.

The results for code are impressive. For Qwen3-Coder-480B and Kimi-K2, Cerebras reports near-lossless compression despite 50 percent pruning. In the vLLM documentation, Qwen3-30B-A3B shrinks from 56.93 to 29.92 GiB and keeps 99.8 percent of its performance on the gsm8k_platinum math test.

But the same sources also show the limits:

• In the REAP paper, Qwen3-30B-A3B with 50 percent pruning drops from 0.721 to 0.503 on average across the multiple-choice knowledge tests. On WildBench, which the paper classifies as creative writing, it falls from 0.811 to 0.718.
• According to the vLLM docs, Moonlight-16B-A3B keeps only 17 percent of its math performance with 50 percent pruning.
• When calibrated on general web text (C4) instead of code, coding performance collapsed, according to the REAP paper. Several models produced nothing but incoherent output.

The vLLM docs explicitly warn that if certain topics or features are missing from the calibration data, the experts responsible for them may be the ones that get removed.

HOPE: experts work as a team

Amazon's HOPE (Higher-Order Pruning of Experts, arXiv 2609.18916, code at github.com/awslabs/HOPE, Apache-2.0) targets a weakness in REAP. REAP scores each expert in isolation, as if contributions simply added up. In practice, experts complement each other, because the router often calls the same pairs together. HOPE captures these pairwise effects in a matrix. Its diagonal roughly matches the REAP scores, and the other entries describe how strongly two experts work together. Choosing which experts to drop is then solved as a quadratic optimization problem: find the combination whose removal increases the estimated error the least. REAP is a special case of HOPE with the interactions left out.

Amazon tested Qwen3.5-122B, Qwen3.5-35B and GLM-4.5-Air against REAP, MAN, EAN and plain frequency counting. At 50 percent pruning, HOPE gets a mean rank of 1.58, with REAP second at 2.42. The widely quoted figure comes from the 35B model: on SWE-Bench Pro (fixing real-world bugs), the halved HOPE version scores 42.3, the original 41.8 and REAP 36.2. For the large model, it's worth checking the appendix:

So the model only stays on par with the original when the calibration data closely matches the target task. On the 122B model with HOPE, just switching between two kinds of code data makes a difference of almost seven points. At 10 to 20 percent pruning, Amazon says REAP and MAN are "sometimes competitive."

For the general tests (the Tulu3 suite), Amazon's blog says code calibration didn't badly damage other capabilities. The paper's tables tell a more mixed story. On the 35B model at 50 percent, the MMLU knowledge test drops from 65.15 to 29.77 with HOPE and to 50.15 with REAP. Our take: some of the original models' scores look implausible. The 122B model, for example, scores 26.68 on MMLU, close to random chance. So these tests are of limited use as a measure of general knowledge. The paper doesn't include a single test in a language other than English.

The blind spot: what gets used for calibration

The Model Cards show the same pattern. 0xSero/Qwen3.5-99B removes 20 percent of the experts (from 256 to 205 per layer, 198 GB in BF16) and reports 97.9 percent retained performance, but measured only on English tests like ARC, HellaSwag and Winogrande. 0xSero/Qwen3.5-76B removes 40 percent (154 experts per layer, 152 GB) and lists no benchmarks at all. The OpenMOSE variant (22 percent removed) at least warns explicitly that performance may be uneven in languages underrepresented in the calibration data.

In January 2026, Cerebras admitted as much itself on Reddit:

"We've gotten a lot of feedback that REAP pruning hurts the model's creative writing and multilingual abilities. […] That's to be expected with our REAPs, given a calibration set curated for agentic coding."

One user in the thread called the loss of multilingual ability "a big loss." The model's Model Card, meanwhile, lists just two code benchmarks, HumanEval and HumanEval+.

What research says about other languages

• Where language lives in the model: Bandarkar et al. (ICLR 2026) found in four MoE models that routing for non-English text differs from English mainly in the first and last layers. In the middle layers it's largely language-agnostic. In the last layers, a small group of experts handles generating non-English text. When those experts were switched off as a test, performance dropped significantly.
• Lithuanian as a warning sign: A study by Hostinger Research from November 2026 (arXiv 2611.07654) pruned four MoE models with REAP, including Qwen3.6-35B-A3B and GLM-4.7-Flash. At 25 percent pruning with English calibration, the median drop in reading comprehension (Belebele) was 1.7 points in English and 21.4 in Lithuanian. The cause is routing: in the last layers, Lithuanian sends 55 to 72 percent of its expert calls to just eight experts, versus 19 to 39 percent for English, and English calibration deletes some of them. Lithuanian calibration data recovered most of the loss without hurting English. At 50 percent, though, even Lithuanian-calibrated models lost 6 to 15 points. Typical failures: the models don't stop writing, switch to English, invent word forms and mangle special characters.
• Counterevidence for major languages: The same study cites work showing that English calibration transfers well to widely spoken languages. Salim et al. (ACL 2026) write that calibration sets in different languages can lead to very similar pruning results. Martin, Bandarkar and Peng (arXiv 2605.28042) managed to extract translation specialists from GPT-OSS-20B and Qwen3-30B-A3B, including for German, removing half the experts without significant loss. But they calibrated on translation data (FLORES-200) and only measured translation. For German, the authors note that uniform pruning across all layers led to unusable output sooner.
• Older forms of pruning: Kurz et al. (TACL) studied pruning individual weights in conventional models that aren't split into experts. For Llama-3 8B, German calibration instead of English gave better German scores, for instance on the Belebele reading test (54.3 instead of 36.6 points, versus 66.1 unpruned). But the authors stress that calibrating in the target language doesn't guarantee good reasoning results.

What this means for German, and how to test it

German is far better represented in training data than Lithuanian. Our take: a 20-point collapse is therefore less likely. But nobody has shown that for REAP or HOPE. We couldn't find any published measurement comparing a pruned model with its original in German, and we don't have a solid measurement of our own for this article. Still, the evidence from Cerebras' admission, the Lithuanian study and the routing research all points the same way: what's most at risk is what happens in the last layers, meaning fluent, idiomatic phrasing. Rare knowledge is at risk too. Code benchmarks won't notice any of it.

A field report from Black Sheep AI (August 2026) shows how subtle this can be, although it used a simpler, frequency-based method. A pruned Qwen3.5-397B translating a sentence into Spanish inserted a Chinese character mid-sentence and then switched to English. The automated check for outright failures had rated the model as fine.

If you want to use a pruned model for German text, run it side by side with the original at the same quantization. Good options are the German portion of Global-MMLU, MGSM-de, Belebele (deu_Latn) and FLORES for translation. Add a few writing tasks of your own, watching for language switching, wrong word forms and endless repetition. According to a GitHub issue, the make-localllm-easier project is planning exactly this kind of per-language comparison, but there are no results yet.

The bigger picture

Expert pruning is a real option for getting huge MoE models onto affordable hardware. Methodologically, HOPE is a sensible step forward because it accounts for experts working as a team, and it does measurably better than REAP at heavy pruning. But the promise of "just as good as the original" only holds for the calibration task: on SWE-Bench Pro, the 122B model's score swings by almost seven points depending on which code data you use. For German, there's no data at all. Neither paper, and none of the community Model Cards we looked at, tests any language other than English.

Our verdict: if you use a model for code and agent tasks in English, a lightly pruned variant (20 to 25 percent) is a safe bet. If you need it to write German, test it yourself before switching. And consider whether more aggressive quantization of the full model is the better route, since that keeps every expert. Black Sheep AI sums it up: "Compress precision, not coverage." German or multilingual calibration sets should become standard for pruned models. According to the Lithuanian study, they barely hurt English.