Strategic Data Ordering: Enhancing Large Language Model Performance through Curriculum Learning (2405.07490v1)

Abstract: The rapid advancement of LLMs has improved text understanding and generation but poses challenges in computational resources. This study proposes a curriculum learning-inspired, data-centric training strategy that begins with simpler tasks and progresses to more complex ones, using criteria such as prompt length, attention scores, and loss values to structure the training data. Experiments with Mistral-7B (Jiang et al., 2023) and Gemma-7B (Team et al., 2024) models demonstrate that curriculum learning slightly improves performance compared to traditional random data shuffling. Notably, we observed that sorting data based on our proposed attention criteria generally led to better performance. This approach offers a sustainable method to enhance LLM performance without increasing model size or dataset volume, addressing scalability challenges in LLM training.

• The paper introduces a curriculum learning strategy that orders LLM training data based on complexity metrics like prompt length, attention scores, and loss values.
• This data ordering approach aims to improve LLM performance on models like Mistral-7B and Gemma-7B without requiring larger models or more data.
• By training on simpler tasks first, the strategy seeks to enhance LLMs' overall performance and potentially reduce computational costs associated with scaling.

This paper introduces a curriculum learning strategy to enhance the performance of LLMs such as Mistral-7B and Gemma-7B, without scaling the model size or dataset volume. The approach involves structuring training data based on complexity, using prompt length, attention scores, and loss values as difficulty metrics. The goal is to improve LLM performance by training the models on simpler tasks before progressively introducing more complex ones.

Here's a more detailed breakdown:

• Introduction: The paper addresses the challenges associated with scaling LLMs, such as increased computational resource demands and costs, and proposes a data-centric training strategy using curriculum learning. This approach is inspired by human educational methods, where