Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?

Investigating the Impact of New Knowledge Introduction on LLMs

This essay provides an in-depth look at the academic paper focused on the consequences of introducing new factual knowledge to LLMs via supervised fine-tuning. The paper aims to assess how such fine-tuning affects a model's ability to utilize its pre-existing knowledge and its propensity to hallucinate inaccurate responses.

Introduction

Pre-training LLMs on vast textual corpora embeds a considerable amount of factual knowledge in their parameters. This knowledge provides a foundation for various downstream applications. However, LLMs often require further alignment through supervised fine-tuning on instruction-following tasks and preference learning from human feedback. This process can introduce new factual information that deviates from the knowledge acquired during pre-training. A prevailing conjecture in the field postulates that exposure to new knowledge during fine-tuning could promote hallucinations, where models generate factually incorrect outputs.

Study Setup and Methodology

To analyze the impact of new knowledge in fine-tuning, the authors designed a controlled setup focused on a closed-book question-answering (QA) task. They categorized the fine-tuning examples into Known and Unknown types, with Known examples further divided into ClearlyKnown, MaybeKnown, and WeaklyKnown categories. The study evaluates how the proportion of Unknown examples in the fine-tuning dataset affects the model's performance and tendency to hallucinate.

Key Findings

1. Learning Dynamics: The study finds that Unknown examples are learned substantially slower than Known examples during fine-tuning. This suggests that LLMs struggle to integrate new factual knowledge through fine-tuning and primarily enhance their ability to utilize pre-existing knowledge.
2. Hallucinations: There is a linear correlation between the proportion of Unknown examples learned and the model's tendency to hallucinate. This highlights the risk of introducing new factual knowledge through fine-tuning, which can compromise the model’s reliability by increasing hallucinations.
3. Overfitting and Early-Stopping: As Unknown examples are primarily learned in the later stages of training, their presence increases the risk of overfitting. The study demonstrates that early-stopping can mitigate this issue, improving development performance by preventing the fitting of most Unknown examples.
4. Filtering Unknown Examples: Removing Unknown examples from the fine-tuning dataset significantly reduces the risk of overfitting without sacrificing performance. This indicates that aligning the fine-tuning data with the model's pre-existing knowledge is crucial for optimal performance.
5. Performance Across Categories: Fine-tuning on ClearlyKnown examples alone does not yield the best results. Incorporating MaybeKnown examples, which represent facts with lower certainty, is essential for handling such examples during inference, thereby improving the model's performance.

Implications for Practice and Theory

The study's findings have several practical implications. Fine-tuning with a high proportion of Unknown examples can degrade model performance and increase hallucinations. Thus, it is advisable to control the introduction of new factual knowledge during fine-tuning. Techniques such as early-stopping and filtering-out Unknown examples can be effective in maintaining model reliability.

From a theoretical perspective, the findings support the hypothesis that LLMs mostly acquire factual knowledge through pre-training, while fine-tuning predominantly teaches models to use this knowledge more efficiently. This underscores the limited efficacy of supervised fine-tuning as a means to integrate new factual knowledge, suggesting a need for alternative methods or refined fine-tuning approaches.

Future Directions

Future research could explore various avenues to address these issues:

• Developing robust methods for filtering or appropriately labeling new factual information encountered during fine-tuning.
• Investigating the long-term effects of new knowledge introduction in broader and more diverse dataset contexts.
• Exploring alternative fine-tuning strategies that can enhance the integration of new knowledge without promoting hallucinations.

Conclusion

The paper provides significant insights into the dynamics of knowledge acquisition in LLMs and the consequences of introducing new factual information through fine-tuning. The results demonstrate that while LLMs enhance their utilization of pre-existing knowledge through fine-tuning, they struggle to integrate new knowledge, which leads to increased hallucinations. Practitioners should consider these findings when designing fine-tuning processes to avoid adverse effects and maximize the efficiency of LLMs.