Mixture‑of‑Kittens (MoK) solves the communication‑bound problem that limits MoE training on large GPU clusters. In today’s Mixture‑of‑Experts workloads the routing and reduction steps can consume more than half of the total training time, leaving tensor cores idle while data moves between GPUs. Teams that rely on separate communication libraries see poor utilization, especially when expert loads are imbalanced, and they still need CPU‑GPU synchronization that adds latency on Grace‑based Blackwell nodes.
MoK fuses every MoE communication and computation step into a single deterministic megakernel that runs entirely on the GPU. It uses pull‑based forward dispatch combined with push‑based forward combine, cutting the signalling overhead from about 103 µs to just 18 µs and raising NVLink lane usage by up to 29 % under skewed expert distribution. A fixed‑size ring token buffer eliminates the need for the CPU to size buffers or drop tokens, removing all CPU‑GPU syncs and guaranteeing zero token loss. Overlap granularity is tuned to the middle ground—targeting at least two full SM waves per expert‑grouped GEMM—so the tensor cores stay busy without excessive barrier stalls.
Benchmarks on a single NVL72 rack show up to 2.37× faster MXFP8 forward and 1.92× BF16 forward versus the best public baseline, translating to a 1.41× end‑to‑end throughput gain on 512 GPUs. The kernel is Apache‑2.0 licensed but requires NVIDIA Blackwell SM100/SM103 GPUs (GB200 or GB300 NVL72 racks), Python 3.12+, PyTorch 2.10+, and CUDA 13.0+. Organizations that own or can rent NVL72 capacity—frontier AI labs, well‑funded model startups, GPU‑focused cloud providers, and national supercomputing centers—can drop MoK into their training stack today to reclaim the compute lost to MoE communication.
#AI #ML #GPUComputing #MoE #DeepLearning #HPC