LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement (2403.15042v2)

Abstract: Pretrained LLMs are currently state-of-the-art for solving the vast majority of natural language processing tasks. While many real-world applications still require fine-tuning to reach satisfactory levels of performance, many of them are in the low-data regime, making fine-tuning challenging. To address this, we propose LLM2LLM, a targeted and iterative data augmentation strategy that uses a teacher LLM to enhance a small seed dataset by augmenting additional data that can be used for fine-tuning on a specific task. LLM2LLM (1) fine-tunes a baseline student LLM on the initial seed data, (2) evaluates and extracts data points that the model gets wrong, and (3) uses a teacher LLM to generate synthetic data based on these incorrect data points, which are then added back into the training data. This approach amplifies the signal from incorrectly predicted data points by the LLM during training and reintegrates them into the dataset to focus on more challenging examples for the LLM. Our results show that LLM2LLM significantly enhances the performance of LLMs in the low-data regime, outperforming both traditional fine-tuning and other data augmentation baselines. LLM2LLM reduces the dependence on labor-intensive data curation and paves the way for more scalable and performant LLM solutions, allowing us to tackle data-constrained domains and tasks. We achieve improvements up to 24.2% on the GSM8K dataset, 32.6% on CaseHOLD, 32.0% on SNIPS, 52.6% on TREC and 39.8% on SST-2 over regular fine-tuning in the low-data regime using a Llama-2-7B student model. Our code is available at https://github.com/SqueezeAILab/LLM2LLM .

• The paper introduces an iterative teacher-student paradigm to synthesize training examples addressing model errors.
• The method achieves performance boosts up to 24.2% on GSM8K and 32.6% on CaseHOLD over baseline approaches.
• The approach minimizes reliance on manual annotations and outperforms existing techniques like EDA and AugGPT.

Enhancing LLMs with LLM2LLM

The paper "LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement" proposes a novel iterative data augmentation technique designed to enhance the performance of LLMs in low-data regimes. By leveraging the strengths of both teacher and student LLMs, this approach addresses a critical bottleneck in fine-tuning LLMs for specific tasks where limited annotated data is available.

Methodology

LLM2LLM utilizes a teacher-student model paradigm to iteratively generate synthetic data. The process begins by fine-tuning a smaller LLM, termed the student model, on an initial seed dataset. Upon completion, the student model is evaluated on this seed dataset, and incorrect predictions are identified. The teacher LLM is then employed to generate new examples similar to these incorrect instances, which are integrated back into the training dataset.

Figure 1: LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement. One iteration of LLM2LLM begins with training and evaluating the model on the training data. Incorrect answers from the training data are used as inputs to generate extra samples with similar styles to the teacher model.

This iterative process of training, evaluating, and augmenting continues, resulting in a progressively refined dataset tailored to the student's deficiencies. Such a strategy is not only effective but also scalable, minimizing the dependency on extensive human-annotated datasets.

Experimental Results

The efficacy of LLM2LLM was evaluated across diverse datasets, including GSM8K and CaseHOLD. Significant improvements were reported, with increases of up to 24.2% on GSM8K and 32.6% on CaseHOLD over baseline fine-tuning approaches. These results underscore the framework's capability to operate effectively in data-constrained environments and efficiently scale task-specific datasets with synthesized data.

Figure 2: LLM2LLM on GSM8K (left) and CaseHOLD (right) with various seed data sizes. Each line shows the test accuracy of the finetuned LLaMA-2-7B model with each step of LLM2LLM with varying seed dataset size.

Comparison with Baselines

LLM2LLM was compared against several baseline augmentation methods, including EDA and AugGPT. The framework consistently outperformed these baselines, demonstrating its ability to generate more relevant and challenging examples for the student model, ultimately leading to superior performance.

Teacher Model Selection

Ablation studies reveal that the choice of teacher model significantly impacts the quality of augmentation. Notable differences in performance were observed when using various teacher LLMs like GPT-3.5, GPT-4-Turbo, and LLaMA2-70B. For example, using GPT-4-Turbo as the teacher model resulted in notably higher accuracy due to its superior reasoning capabilities.

Practical Implications and Future Directions

LLM2LLM provides a robust framework for enhancing LLMs in low-resource settings, reducing the need for labor-intensive data curation. Its iterative and targeted approach ensures that generated data addresses specific weaknesses of the student model, making it an invaluable tool for domain-specific LLM applications.

Future research could explore tuning hyperparameters within this framework or integrating it with complementary AI techniques, such as prompt tuning or few-shot learning. Additionally, examining the application of LLM2LLM to other modalities and tasks beyond NLP could further extend its utility.

Conclusion

LLM2LLM presents an effective strategy for augmenting training datasets in data-scarce scenarios, using iterative LLM-based data generation. Its deployment can substantially enhance task-specific LLMs, accelerating their adaptation and performance in specialized domains. Such advancements pave the way for more accessible and efficient AI solutions across various applications.