Prime‑RL 0.6.0 solves the core pain points that teams face when trying to train trillion‑parameter Mixture‑of‑Experts models on long‑horizon, agentic workloads such as software‑engineering agents. The biggest frustration is idle GPU time caused by waiting for extremely long rollouts before each policy update. By decoupling trainer and inference, Prime‑RL lets the optimizer push new weights as soon as they are ready while already‑dispatched rollouts continue with their cached prefix. This eliminates the synchronization bottleneck and keeps GPUs busy.
Another common headache is the mismatch between what the trainer sees and what inference actually serves, which silently degrades learning stability. Router replay (R3) captures the exact routing decisions made during inference and feeds them back to the trainer, cutting KL divergence by roughly an order of magnitude. Pairing this with FP8 training—using block‑scaled precision from DeepSeek V3—aligns trainer and inference numerics, further stabilizing the loop.
Inference itself often becomes the throughput ceiling, especially when prefill and decode stages compete for the same workers. Prime‑RL separates prefill and decode pipelines, spreads experts across ≥ 32 GPUs via Wide Expert Parallelism, and runs inference in FP8 with DeepEP/DeepGEMM kernels. KV‑cache pressure is handled by tiered offloading to CPU and disk or a centralized Mooncake store, letting the system scale to massive concurrent rollouts without blowing memory.
For training, the framework builds on TorchTitan and adds three‑dimensional parallelism: FSDP shards parameters, gradients and optimizer states; Expert Parallelism splits the massive expert layers across nodes; Context Parallelism tackles activation memory at 130k+ sequence lengths. Together these tricks let a GLM‑5‑scale model finish a full RL run on just 28 H200 nodes with step times under five minutes.
In short, teams struggling with long rollout latency, trainer‑inference drift, and inference bottlenecks can now run large‑scale agentic RL more efficiently, with predictable step times and stable learning, all from a single Slurm command.
#AI #MachineLearning #ReinforcementLearning #LLMOps #MLEngineering #Productivity