MoonEP tackles a core bottleneck in scaling Mixture‑of‑Experts models: routing imbalance. In expert parallelism each token is sent to its top‑K experts, but routers rarely distribute work evenly. Some experts receive far more tokens than others, making the slowest rank dictate iteration time. This imbalance causes variable activation shapes, GPU memory fragmentation, and per‑layer host synchronization that erodes throughput and can even trigger out‑of‑memory failures at scale.
MoonEP’s answer is a simple invariant: every expert‑parallel rank always receives exactly S × K tokens, regardless of how skewed the router is. It achieves this by dynamically planning a small set of redundant experts directly from the current router outputs, prefetching them before expert computation, and reducing their gradients back to their home ranks during the backward pass. Three properties make this practical:
- Perfect balance – the S × K guarantee is enforced online, eliminating the slowest‑rank effect.
- Online planning – a lightweight GPU kernel written in CUTLASS CuTe computes the redundant expert plan with negligible overhead.
- Zero copy and static shapes – fused permute/unpermute writes tokens directly into expert‑grouped buffers on remote ranks; the buffer view is returned to computation, so only a fixed S × K buffer is needed and host synchronization per MoE layer disappears.
The result is flat communication cost as imbalance grows, stable memory usage, and removed per‑layer synchronization headaches. Teams training or serving trillion‑parameter MoE models can now scale expert parallelism without rewriting their routing logic or over‑provisioning GPU memory. MoonEP ships under an MIT license, integrates with existing frameworks, and delivers the 2.5× scaling efficiency boost reported for Moonshot’s Kimi K3 model.