Researchers working on agentic reinforcement learning spend too much time wrestling with framework plumbing rather than experimenting with algorithms. Every new estimator, rollout scheme, or pipeline stage forces changes across trainer code, distributed backends, and glue logic, multiplying iteration cost.
Molt from NVIDIA’s NeMo team solves this by keeping the entire RL codebase small enough to hold in a researcher’s head—about 8.6 K lines versus tens of thousands in other frameworks. It is PyTorch‑native and built from three unchanged upstream components: Ray for placement and async queues, vLLM for rollout, and NeMo AutoModel with FSDP2 for training. Because none of these pieces are forked, upstream improvements arrive simply by updating the container pin.
The runtime consists of an agent pool, a router of vLLM engines, and a single trainable policy actor. A streaming pool keeps prompt groups in flight so engines never idle while the actor trains. When a weight update is ready, Molt broadcasts the new actor shards directly to each rollout engine over NCCL, bypassing the router and preserving in‑flight requests.
Agents are ordinary Python programs: a single module exposing an AgentRunner, with reward logic written as any Python function. Two interaction styles are supported—Env mode where Molt runs the LLM loop, or ChatAgent mode where the user drives the loop via standard OpenAI or Anthropic SDKs through a loopback server that token‑exactly captures prompts and completions.
Three correctness invariants guarantee that training only sees tokens that were actually sampled, preserving token identity and policy‑version semantics even for mixture‑of‑expert models via rollout‑routing replay. This eliminates that class of silent, biased updates that plague other RL pipelines.
With the provided Slurm scripts, pre‑built container, and example recipes, teams that have access to two nodes of eight H100 GPUs can start multi‑turn tool‑use agents, code‑execution agents, vision‑language environments, LLM‑as‑judge reward loops, or on‑policy distillation without re‑architecting their code.
#AI #ML #ReinforcementLearning #RL #AgenticAI #Molt