NVIDIA AI Introduces AceReason-Nemotron: Enhancing Math and Code Reasoning with Reinforcement Learning
Introduction
Reasoning is a critical component of advanced AI systems. The launch of OpenAI’s o1 sparked interest in developing reasoning models using large-scale reinforcement learning (RL). However, the initial release of DeepSeek-R1 lacked crucial technical details, such as data curation strategies and specific RL training methods. This absence has resulted in fragmented research efforts and challenges in replicating findings.
Challenges in Current Approaches
Training language models for reasoning in mathematics and coding usually involves pretraining and supervised fine-tuning. Early RL attempts with domain-specific reward models faced obstacles due to the complexities of math and coding tasks. Although recent methods have incorporated rule-based verification, they often focus on a single domain and lack thorough benchmark evaluations, which can affect training stability.
NVIDIA’s Innovative Approach
NVIDIA researchers have shown that large-scale RL can significantly improve the reasoning capabilities of small- and mid-sized models. Their approach includes a sequential training strategy that first focuses on math-only prompts and then on code-only prompts. This method has demonstrated that training with math-only RL not only enhances performance in math but also positively impacts coding tasks. Further iterations of code-only RL have been shown to improve code performance without compromising math results.
Data Curation Pipeline
A comprehensive data curation pipeline has been established to gather challenging prompts with high-quality, verifiable answers and test cases. This pipeline combines the DeepScaler and NuminaMath datasets for math, covering various topics such as algebra and geometry, while rigorously filtering out unsuitable content. For coding, datasets are sourced from competitive programming platforms, ensuring a wide range of test cases, including edge cases.
Performance Outcomes
The AceReason-Nemotron-7B model achieved impressive accuracy improvements, with a 14.5% and 14.6% increase on AIME 2024/2025, and a 14.2% and 8% boost on LiveCodeBench v5/v6 compared to initial supervised fine-tuning models. The 14B variant outperformed larger models like DeepSeek-R1-Distill-Qwen-32B and DeepSeek-R1-Distill-Llama-70B, establishing itself as a leader among open RL-based reasoning models. Notably, AceReason-Nemotron-14B surpassed OpenMath-14B/32B on AIME benchmarks and outperformed OpenCodeReasoning-14B on LiveCodeBench.
Conclusion
In conclusion, research indicates that large-scale RL significantly enhances the reasoning capabilities of small- and mid-sized supervised fine-tuning models. The sequential training approach, beginning with math and followed by code, demonstrates that focusing on mathematical reasoning can improve overall performance across both domains. The robust data curation pipeline supports verification-based RL, highlighting its effectiveness in advancing model reasoning and setting new performance standards.
Further Reading
For more insights, check out the research paper and model on Hugging Face. Acknowledgments go to the researchers involved in this project. Stay connected with us on Twitter, join our 95k+ ML SubReddit, and subscribe to our newsletter.
Transforming Your Business with AI
- Explore how AI technology can enhance your work processes.
- Identify areas for automation and customer interactions where AI can add value.
- Establish key performance indicators (KPIs) to measure the impact of your AI investments.
- Choose customizable tools that align with your business objectives.
- Start with a small project, assess its effectiveness, and gradually expand your AI initiatives.
If you need assistance in managing AI in your business, feel free to contact us at hello@itinai.ru or reach us on Telegram, X, and LinkedIn.
