Enhancing LLM Reasoning with Multi-Attempt Reinforcement Learning

Enhancing LLM Reasoning with Multi-Attempt Reinforcement Learning

Recent advancements in reinforcement learning (RL) for large language models (LLMs), such as DeepSeek R1, show that even simple question-answering tasks can significantly improve reasoning capabilities. Traditional RL methods often focus on single-turn tasks, rewarding models based solely on the correctness of one response. However, these methods face challenges like sparse rewards and do not effectively train models to refine their answers based on user feedback. To overcome these limitations, multi-turn RL approaches have been developed, allowing LLMs to make several attempts at solving a problem, thereby enhancing their reasoning and self-correction skills.

Exploration of Planning and Self-Correction

Several studies have examined planning and self-correction mechanisms in RL for LLMs. Some approaches, inspired by the Thinker algorithm, allow agents to explore alternatives before taking action, which enhances reasoning by enabling multiple attempts rather than creating a world model. Techniques like SCoRe train LLMs on multi-attempt tasks but often lack verification of prior responses using ground-truth rewards, leading to complex calibration. Other methods employ external tools for self-correction, such as Reflexion for self-reflection and CRITIC for real-time feedback. The proposed method builds on DeepSeek R1’s single-turn question-answering task by introducing a multi-attempt framework that utilizes historical errors to refine responses and improve reasoning.

Multi-Attempt RL Approach

Researchers from DualityRL and Shanghai AI Lab have introduced a multi-attempt RL approach to enhance reasoning in LLMs. Unlike single-turn tasks, this method allows models to refine their responses through multiple attempts with feedback. Experimental results indicate a significant accuracy improvement of 45.6% to 52.5% with two attempts on mathematical benchmarks, compared to minimal gains in single-turn models. The model learns self-correction using Proximal Policy Optimization (PPO), leading to enhanced reasoning capabilities. This multi-attempt setting supports iterative refinement, promoting deeper learning and problem-solving skills, making it a promising alternative to traditional RLHF and supervised fine-tuning methods.

Iterative Refinement Process

In a single-turn task, an LLM generates a response to a question from a dataset, optimizing its policy to maximize rewards based on answer correctness. In contrast, the multi-turn approach allows for iterative refinement, where responses influence subsequent prompts. The proposed multi-attempt task introduces a fixed number of attempts, prompting retries if the initial response is incorrect. The model receives a reward of +1 for correct answers, -0.5 for incorrect but well-formatted responses, and -1 for otherwise. This strategy encourages exploration in early attempts without penalties, using PPO for optimization and enhancing reasoning through reinforcement learning.

Training and Results

The study fine-tunes the Qwen 2.5 Math 1.5B model on 8,000 math questions using PPO with specific parameters. Training spans 160 episodes, generating 1.28 million samples. In the multi-attempt setting, attempts are sampled from 1 to 5, while the baseline follows a single-turn approach. Results indicate that the multi-attempt model achieves higher rewards and slightly better evaluation accuracy, improving response accuracy from 45.58% to 53.82% over multiple attempts. This adaptive reasoning capability could enhance performance in code generation and problem-solving fields.

Conclusion

This study builds on DeepSeek R1’s question-answering task by introducing a multi-attempt mechanism. While performance gains on math benchmarks are modest, the approach significantly enhances the model’s ability to refine responses based on feedback. By training the model to iterate on incorrect answers, search efficiency and self-correction improve. Experimental results show accuracy increases from 45.6% to 52.5% with two attempts, whereas a single-turn model shows only slight improvement. Future research could explore incorporating detailed feedback or auxiliary tasks to further enhance LLM capabilities, making this approach valuable for adaptive reasoning and complex problem-solving tasks.

Transforming Your Business with AI

Explore how artificial intelligence technology can transform your approach to work, such as enhancing LLM reasoning with multi-attempt reinforcement learning. Look for processes that can be automated and identify customer interactions where AI can add the most value. Establish important KPIs to ensure your AI investment positively impacts your business. Choose tools that meet your needs and allow customization to achieve your objectives. Start with a small project, gather data on its effectiveness, and gradually expand your use of AI.

If you need guidance on managing AI in business, contact us at hello@itinai.ru. Follow us on Telegram, X, and LinkedIn.