Practical Solutions for Training Large Language Models (LLMs)
Enhancing Model Performance with Compute-Efficient Synthetic Data
A critical challenge in training large language models (LLMs) for reasoning tasks is identifying the most compute-efficient method for generating synthetic data that enhances model performance.
Traditionally, stronger and more expensive language models (SE models) have been relied upon to produce high-quality synthetic data for fine-tuning. However, this approach is resource-intensive and restricts the amount of data that can be generated within a fixed computing budget.
Current methods for improving LLM reasoning capabilities include strategies such as knowledge distillation and self-improvement, which have proven effective but come with significant drawbacks, such as high computational costs that limit the volume and diversity of data produced.
The researchers from Google DeepMind introduce a novel approach that challenges the reliance on SE models for synthetic data generation. They advocate for using weaker but cheaper models (WC models), which, despite their lower quality, are more cost-effective and enable the generation of larger data volumes within the same computing budget.
The technical details involve a comparative analysis between SE and WC models under a fixed compute budget. Experiments were conducted using the Gemma2 family of models on datasets like MATH and GSM-8K, with Gemma2-9B and Gemma2-27B representing WC and SE models, respectively.
Significant improvements in LLM performance were observed across various benchmarks. Fine-tuning models on data generated by WC models consistently yielded better results than those trained on data from SE models.
Using WC models for synthetic data generation proves to be more compute-efficient than relying on SE models. By generating more diverse and comprehensive training data within a fixed compute budget, WC models enable the training of stronger LLM reasoners.
Discover how AI can redefine your way of work. Identify Automation Opportunities, Define KPIs, Select an AI Solution, Implement Gradually. For AI KPI management advice, connect with us at hello@itinai.com.
Discover how AI can redefine your sales processes and customer engagement. Explore solutions at itinai.com.
