Introduction: Reinforcement Learning Progress through Chain-of-Thought Prompting
Large Language Models (LLMs) have made remarkable strides in tackling complex reasoning tasks, largely due to the innovative approach of Chain-of-Thought (CoT) prompting combined with large-scale reinforcement learning (RL). Notable models like Deepseek-R1-Zero have showcased impressive reasoning abilities by directly applying RL to base models. Other methods, including SimpleRL and Open-ReasonerZero, have demonstrated enhancements in smaller models, such as those in the Qwen series. However, achieving consistent success across various base model families remains a significant hurdle. The challenge of applying R1-Zero-style training to models like the Llama series raises critical questions about the differing behaviors observed during reinforcement learning.
Limitations of RL Scaling on Llama Models
While advancements in large-scale RL have been observed in models such as OpenAI’s o1 and o3, and DeepSeek’s R1, there is an ongoing interest in exploring RL applications on smaller models with fewer than 100 billion parameters. However, efforts have primarily focused on the Qwen model family, making it difficult to replicate results across families like Llama. A lack of transparency in pre-training pipelines complicates our understanding of how pre-training influences RL scaling. Some unconventional studies suggest that one-shot prompting can enhance reasoning in Qwen models but offers limited benefits for Llama models. Initiatives like OpenWebMath and MathPile have made progress in curating high-quality mathematical pre-training corpora, yet they still face constraints in scale, particularly under 100 billion tokens.
Exploring Mid-Training with Stable-then-Decay Strategy
Researchers at Shanghai Jiao Tong University have delved into how mid-training strategies can influence RL dynamics, particularly concerning Qwen and Llama models. Their study yielded several key findings:
- High-quality mathematical corpora, such as MegaMath-Web-Pro, significantly enhance both base model and RL outcomes.
- QA-style data, especially with extensive CoT reasoning, further improves RL results.
- Long CoT prompts can lead to verbosity and instability during RL training.
- Implementing scaling during mid-training has been shown to enhance downstream RL performance.
To address these findings, researchers introduced a two-stage mid-training strategy called Stable-then-Decay. This involves training base models on 200 billion tokens, followed by 20 billion tokens across three CoT-focused branches. This innovative approach led to the creation of OctoThinker models, which demonstrate strong compatibility with RL.
RL Configuration and Benchmark Evaluation
The MATH8K dataset served as the foundation for RL training prompts, with a configuration that included a global training batch size of 128, 16 rollout responses per query, and a PPO mini-batch size of 64. Experiments were conducted on Llama-3.2-3B-Base and Qwen2.5-3B-Base models. Evaluation utilized few-shot prompting for base language models and zero-shot for RL-tuned models across various indicator tasks, including GSM8K, MATH500, OlympiadBench, and AMC23. During RL training, Qwen models exhibited increasing response lengths that remained within reasonable limits, while Llama showed abnormal behavior, with average response lengths soaring to 4,096 tokens. Evaluation results indicated that the RL-tuned Qwen2.5-3B achieved improvements across benchmarks, while the Llama-3.2-3B demonstrated only marginal gains.
OctoThinker Outperforms Llama in RL Compatibility
Each branch of the OctoThinker showed a 10%-20% improvement over the original Llama base model, consistently outperforming the stable-stage model across all sizes when assessed on 13 mathematical benchmarks. The OctoThinker-Zero families revealed varied thinking behaviors during RL scaling, with the OctoThinker-Long variant displaying particularly strong performance. In comparisons among three 3B-scale base models during RL training, the OctoThinker-Long-3B surpassed the original Llama-3.2-3B model and achieved performance parity with Qwen2.5-3B, known for its robust reasoning capabilities. The hybrid and short branches, however, exhibited slightly lower performance, especially on more challenging benchmarks.
Conclusion and Future Work: Toward RL-Ready Foundation Models
This research sheds light on the reasons behind the differing behaviors of base models like Llama and Qwen during RL for reasoning tasks. It emphasizes the crucial role of mid-training in enhancing RL scalability. The two-stage mid-training strategy effectively transforms Llama into a foundation model that is more compatible with RL, culminating in the development of the OctoThinker models. Future research directions include:
- Curating higher-quality mathematical corpora to improve mid-training.
- Creating RL-friendly base models using open recipes without relying on distillation from long CoT reasoning models.
- Separating the QA format and content to assess their individual contributions.
- Expanding the OctoThinker family with new branches, such as tool-integrated reasoning.
FAQ
- What is Chain-of-Thought prompting? It’s a technique that enhances the reasoning capabilities of language models by encouraging them to articulate their thought processes.
- How does reinforcement learning improve language models? RL helps models learn from feedback, allowing them to optimize their responses and improve their performance on various tasks.
- What are the limitations of Llama models in RL? Llama models have shown inconsistent performance in RL settings, particularly when compared to models like Qwen.
- What is the Stable-then-Decay strategy? It’s a two-stage mid-training approach that involves extensive initial training followed by focused training on specific tasks, aimed at improving RL outcomes.
- What are the future directions for OctoThinker models? Future work includes enhancing mathematical corpora, developing new RL-friendly models, and expanding the OctoThinker family with additional features.
In summary, the research from Shanghai Jiao Tong University provides valuable insights into the dynamics of reinforcement learning in large language models. By understanding the role of mid-training and implementing innovative strategies like OctoThinker, we can pave the way for more robust and capable foundation models that excel in reasoning tasks.
