Large hybrid MoE models such as Nemotron-3-Super give high accuracy but are costly to serve. Their active parameters, KV cache and Mamba state limit how many users a single node can handle at a given token‑per‑second rate. This creates two practical problems: low throughput on multi‑GPU servers and inability to run long‑context requests concurrently on a single H100.
NVIDIA’s Nemotron‑Labs-3-Puzzle-75B-A9B addresses these issues. It keeps the original 88‑block hybrid layout but reduces total parameters from 120.7 B to 75.3 B and active parameters from 12.8 B to 9.3 B. The Mamba SSM state shrinks from 128 to 96, and routed‑expert intermediate size is cut to roughly 60 % of the original, halving the activated expert count per token. Because attention layers are untouched, KV‑cache cost per request stays the same.
The compression yields clear performance gains. On an 8×B200 node the model delivers 1.6‑2.14× higher total throughput at matched user‑token rates, with the biggest uplift in decode‑heavy workloads. On a single H100 the weight footprint drops from ~70 GB to ~44.5 GB (NVFP4), freeing enough memory to run eight 1‑million‑token requests instead of one, raising aggregate decode throughput roughly four‑fold.
Accuracy losses are modest and task‑dependent. Arena‑Hard‑V2 drops 4.2 points and SWE‑Bench falls 2.6 points, while long‑context benchmarks like RULER stay within 1‑2 points. The iterative Puzzle search combined with staged knowledge distillation recovers most of the lost quality, and post‑training quantization (FP8 for Hopper, NVFP4 for Blackwell) adds little extra degradation.
Teams can now deploy the compressed model for ultra‑long‑context RAG on a single GPU, scale interactive coding assistants to double the request rate, or run agentic software‑engineering loops with only a small accuracy trade‑off. The approach shows how targeted pruning, iterative search, and mixed‑precision quantization turn a costly MoE into a practical, high‑throughput service.
#AI #Product #LLM #Deployment #Performance #Efficiency