PLMs have transformed Natural Language Processing, but their computational and memory needs pose challenges. The authors propose LoftQ, a quantization framework for pre-trained models. They combine low-rank approximation and quantization to approximate high-precision weights. Results show LoftQ outperforms QLoRA in various tasks, with improved performance in Rouge-1 for XSum and CNN/DailyMail using 4-bit quantization. Further advancements are expected to enhance PLMs’ practical deployment.
Meet LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models
The introduction of Pre-trained Language Models (PLMs) has revolutionized Natural Language Processing (NLP). These models have shown exceptional proficiency in various language tasks such as Natural Language Understanding (NLU) and Natural Language Generation (NLG). However, the computational and memory requirements of these models pose significant challenges.
In this paper, the authors present a novel quantization framework called LoftQ, specifically designed for pre-trained models that require quantization and LoRA fine-tuning. LoftQ combines low-rank approximation and quantization to approximate the original high-precision pre-trained weights.
Quantization Methods
LoftQ is compatible with different quantization functions, including:
- Uniform quantization: A classic method that divides a continuous interval into categories and stores a local maximum absolute value for dequantization.
- NF4 and NF2: Quantization methods used in QLoRA. They map high-precision values to discrete slots based on a Gaussian distribution.
Through extensive experiments, the authors demonstrate that LoftQ consistently outperforms QLoRA across all precision levels. For example, with 4-bit quantization, they achieve a 1.1 and 0.8 improvement in Rouge-1 for XSum and CNN/DailyMail, respectively.
As the field of NLP advances, LoftQ and similar innovations will help bridge the gap between the potential of PLMs and their practical deployment, benefiting a wide range of applications and users.
If you want to evolve your company with AI and stay competitive, consider using LoftQ. It can redefine your way of work by automating customer interactions and improving business outcomes. Connect with us at hello@itinai.com for AI KPI management advice and visit itinai.com to explore practical AI solutions.
