Large hybrid MoE models such as Nemotron‑3‑Super deliver strong accuracy but are costly to serve because their active parameters, KV cache and Mamba state limit how many users a single GPU can handle at a given token rate. NVIDIA’s Nemotron‑Labs‑3‑Puzzle‑75B‑A9B tackles this problem by compressing the parent model while preserving its 88‑block hybrid layout. Total parameters fall from 120.7 B to 75.3 B and active parameters from 12.8 B to 9.3 B, a reduction of about 38 % in total size and 27 % in active compute. The Mamba SSM state shrinks from 128 to 96 channels and the routed‑expert intermediate width is cut to roughly 60 % of the original, which drops the model’s weight footprint from ~70 GB to ~44.5 GB on an H100.
The practical impact is clear. On an 8×B200 node the compressed model achieves 1.60× to 2.14× higher total throughput compared with Super at the same per‑user token floor, with the biggest gains in decode‑heavy workloads (8K input / 64K output). On a single H100 serving 1‑million‑token requests, the memory bound moves from compute to KV cache, allowing concurrency to rise from 1 request to 8 requests, which translates to roughly 4× aggregate decode throughput and a 1.2× faster prefill of long prompts.
Accuracy trade‑offs are modest but noticeable. In BF16 the model loses 1.4 points on MMLU‑Pro, 2.5 on AIME25, 1.9 on GPQA, 1.0 on LiveCodeBench, 1.7 on SciCode, 2.6 on SWE‑Bench and 4.2 on Arena‑Hard‑V2. AA‑LCR and RULER stay within ±0.2 points, showing that long‑context reasoning is largely preserved. The NVFP4 quantization adds little extra loss; on RULER‑1M the NVFP4 checkpoint even scores slightly higher than the BF16 baseline.
Teams should adopt Puzzle‑75B‑A9B for ultra‑long‑context RAG or document analysis on a single GPU, for interactive coding assistants where throughput matters more than a few points of accuracy, and for prefill‑heavy pipelines only if the modest throughput gain justifies the small accuracy dip. For agentic software‑engineering loops, check the SWE‑Bench gap against your task mix; the RL‑focused recovery only partially restores this capability.
In short, if your bottleneck is GPU memory or per‑user token rate and you can tolerate a 1‑4 point accuracy drop on certain benchmarks, switching to Nemotron‑Labs‑3‑Puzzle‑75B‑A9B gives you up to double the serving capacity and up to eight‑fold concurrency on existing hardware.
#AI #LLM #MoE #ModelCompression #Inference #H100