ETH Zurich and HKU Resolve MoE Routing Exploration Dilemma with MirrorMoE: Adaptive Suppression Counteracts Shared Expert Homogenization

Background and the Problem

Modern Mixture-of-Experts (MoE) architectures increasingly pair two dominant design choices: fine-grained expert segmentation (splitting layers into 64 or 128 smaller routed experts) and shared experts (activating dedicated experts unconditionally for every token). In arXiv:2610.22353, researchers from ETH Zurich and HKU revealed an unexpected 'Exploration Dilemma': while shared experts provide general knowledge, they significantly reduce the variance of token representations. This homogenization reduces routing entropy and discourages routers from exploring the exponentially enlarged combinatorial space created by fine-grained segmentation, capping the realized expressiveness of the model.

Architecture and How It Works

1. Token-Adaptive Expert Suppression (MirrorMoE):

• MirrorMoE introduces an adaptive suppression mechanism that dynamically prunes unpromising expert candidates on a per-token basis. By shrinking the local search space adaptively, it forces the router to explore nuanced distinctions among the remaining specialized experts;

1. Zero Training Latency and Computational Overhead:

• Designed with symmetric algebraic projections, MirrorMoE requires no iterative forward-backward passes or complex auxiliary regularization losses, matching or even slightly improving raw per-step iteration runtime compared to vanilla baselines.

Benchmarks and Measured Results

• Pretraining Perplexity Across Scales (Table 1):
• MoE-100M-A38M (8/64/2): Reduces PPL from 54.78 to 51.77 (-3.01 PPL) with 7,844 ms vs 7,871 ms step time;
• MoE-498M-A121M (8/64/2): Cuts PPL from 28.31 to 27.36 (-0.95 PPL);
• MoE-625M-A100M (8/128/2): Drops PPL from 28.46 to 27.84 (-0.62 PPL);
• MoE-2.4B-A270M (8/128/2): Slashes PPL from 21.17 to 20.50 (-0.67 PPL) while lowering step time from 25,711 ms to 25,635 ms.
• Downstream Zero-Shot Evaluations (Table 2):
• Across ARC-E, OBQA, HellaSwag, PIQA, BoolQ, and SIQA, MirrorMoE delivers consistent gains. On MoE-2.4B-A270M, the downstream average score rises from 45.23% to 46.87% (+1.64 pts), led by a +4.03 percentage point surge on BoolQ (58.53% vs 54.50%) and +1.26 pts on ARC-E (46.63% vs 45.37%).

Getting Started for Developers

• Open-Source Implementation: The code is available at https://github.com/BingcongLi/MirrorMoE.
• Integration Strategy: For teams training or fine-tuning sparse MoE models with shared experts, substituting the naive top-k gating with MirrorMoE's token-adaptive suppression operator reconciles exploration and exploitation without increasing step latency or memory footprint.