Scaling Laws for Forgetting When Fine-Tuning Large Language Models (2401.05605v1)

Abstract: We study and quantify the problem of forgetting when fine-tuning pre-trained LLMs on a downstream task. We find that parameter-efficient fine-tuning (PEFT) strategies, such as Low-Rank Adapters (LoRA), still suffer from catastrophic forgetting. In particular, we identify a strong inverse linear relationship between the fine-tuning performance and the amount of forgetting when fine-tuning LLMs with LoRA. We further obtain precise scaling laws that show forgetting increases as a shifted power law in the number of parameters fine-tuned and the number of update steps. We also examine the impact of forgetting on knowledge, reasoning, and the safety guardrails trained into Llama 2 7B chat. Our study suggests that forgetting cannot be avoided through early stopping or by varying the number of parameters fine-tuned. We believe this opens up an important safety-critical direction for future research to evaluate and develop fine-tuning schemes which mitigate forgetting

• The paper finds an inverse linear relationship between fine-tuning performance and forgetting, quantifying the loss of pre-trained knowledge.
• It introduces a novel cross-entropy metric to rigorously evaluate knowledge retention during fine-tuning.
• Scaling behaviors follow shifted power law relationships, guiding effective strategies to balance performance and memory retention in LLMs.

Scaling Laws for Forgetting When Fine-Tuning LLMs

This essay provides a detailed examination of the empirical investigation into the phenomenon of forgetting in LLMs during the fine-tuning phase. The paper "Scaling Laws for Forgetting When Fine-Tuning LLMs" (2401.05605) offers rigorous analysis and results that underscore the intricate interplay between fine-tuning performance, parameter count, and the forgetting effect in LLMs. Key aspects of the investigation, including methodological approaches, empirical results, and implications for future studies in artificial intelligence, are discussed below.

Introduction to Forgetting in Fine-Tuning

The paper addresses catastrophic forgetting, a challenge where LLMs lose previously acquired knowledge during the fine-tuning process, especially when fine-tuning on tasks that require knowledge not represented in its pre-training data. This is particularly concerning in safety-critical applications where models need to retain essential capabilities. The investigation revealed that even parameter-efficient fine-tuning (PEFT) methods such as Low-Rank Adapters (LoRA) are susceptible to significant forgetting.

Figure 1: Generation examples of the pre-trained model, and a model fine-tuned with LoRA, showing updated knowledge, forgotten knowledge, and forgotten alignment behavior.

Empirical Analysis of Forgetting

Inverse Linear Relationship

The core finding of the paper is an inverse linear correlation between the fine-tuning performance (quantified as loss) and the extent of forgetting. This inverse relationship suggests that achieving high fine-tuning performance typically results in greater forgetting of the pre-trained capabilities, aligning with the dataset-specific knowledge accrued during the pre-training phase.

Figure 2: Fine-tuning performance vs Forgetting on OpenOrca and News datasets, illustrating the linear relationship and its implications.

Forgetting Metrics and Evaluation

To quantify forgetting rigorously, the paper introduces a metric based on the cross-entropy between the predictions of the pre-trained model and the fine-tuned model, providing a robust measure that accounts for prior knowledge and prediction shifts. This metric discerns forgetting from mere output format or task performance changes, offering a more precise evaluation of knowledge retention.

Power Law Scaling

In accordance with prior research on the scaling laws in LLMs, this paper finds that both forgetting and fine-tuning loss adhere to power law relationships with the number of parameters fine-tuned and the number of training steps. This is modeled as shifted power law functions, highlighting that increased parameter tuning and training steps exacerbate forgetting. These relationships suggest a fundamental scaling law that guides the optimization of fine-tuning strategies.

Observation of Forgetting Effects

The observed generative behaviors of fine-tuned models provide concrete evidence of forgetting effects. For instance, fine-tuning the LLM on domains such as "OpenOrca" or recent news datasets demonstrated significant forgetting of domain knowledge and reasoning capabilities previously exhibited by the models.

Figure 3: Forgetting and fine-tuning loss trajectories and fit curves for varying ranks and datasets, showcasing empirical fit and trends.

Implications and Future Directions

The implications of these findings are manifold. Firstly, they underscore the necessity for developing mitigation strategies that can balance fine-tuning performance with retention of critical pre-trained capabilities. Strategies could include hybrid optimization algorithms that refine PEFT methods or dynamically adjust the fine-tuning process based on real-time forgetting metrics.

Furthermore, the paper opens avenues for a more detailed exploration of how specific types of knowledge (e.g., reasoning versus safety protocols) are affected by different fine-tuning regimes, informing targeted adjustments in fine-tuning methodologies to preserve core competencies of LLMs.

Conclusion

This investigation extends the understanding of scaling laws within the context of LLM fine-tuning, emphasizing the interplay between model performance, parameter interactions, and the forgetting phenomenon. By establishing a quantitative framework for measuring and analyzing forgetting, the research delineates a new trajectory for enhancing the robustness and reliability of AI applications. The results, striking in their consistency, serve as a pivotal reference point for subsequent empirical and theoretical studies aimed at mitigating forgetting in LLMs.