Audex (Nemotron‑Labs‑Audex‑30B‑A3B) solves a common pain point for teams building multimodal AI: adding audio capabilities often hurts text performance. Most models suffer a “text tax” when they learn to understand or generate sound, causing drops on reasoning and language benchmarks. Audex avoids this regression by keeping a pure‑text backbone (Nemotron‑Cascade‑2‑30B‑A3B) and projecting audio into the same token space as text. The model is a single Mixture‑of‑Experts Transformer with 30 billion total parameters and only 3 billion activated per token, so it runs on standard LLM stacks such as Megatron‑LM for training and vLLM for inference.
Key practical benefits:
– One checkpoint handles audio input and output for speech recognition, translation, text‑to‑speech, text‑to‑audio, and audio understanding without switching models.
– Text scores stay essentially unchanged (MMLU‑Redux 86.4 vs backbone 86.3) and even improve on some reasoning tasks (IMO AnswerBench 81.1 vs 79.3).
– Audio generation uses two codecs: X‑Codec2 for speech (50 tokens/sec) and X‑Codec for general sound (200 tokens/sec), letting the model produce richer non‑speech audio than most open alternatives.
– Training follows a multi‑stage SFT curriculum (text SFT → audio warmup → audio generation → audio understanding) plus text‑only Cascade RL, which preserves and sometimes boosts text abilities while adding audio skills.
– The model supports a 1 million‑token context, enabling long‑form audio‑text tasks.
Limitations to note: the release is under a non‑commercial NVIDIA OneWay license, audio understanding still lags behind top specialist models on MMAR and MMSU, and speech‑to‑speech relies on a cascaded pipeline rather than native full‑duplex processing. Reinforcement learning is currently text‑only, with audio‑text RL slated for future work.
For developers who need a single, deployable model that keeps strong language reasoning while adding flexible audio input and output—whether for call‑center transcription, accessibility tools, sound‑design prototyping, or voice assistants—Audex offers a ready‑to‑use solution that sidesteps the usual trade‑off between modalities.
#AI #Product #MachineLearning #Multimodal #Audio #LLM