Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge (2403.01432v5)

Abstract: LLMs (LMs) memorize a vast amount of factual knowledge, exhibiting strong performance across diverse tasks and domains. However, it has been observed that the performance diminishes when dealing with less-popular or low-frequency concepts and entities, for example in domain specific applications. The two prominent approaches to enhance the performance of LMs on low-frequent topics are: Retrieval Augmented Generation (RAG) and fine-tuning (FT) over synthetic data. This paper explores and evaluates the impact of RAG and FT on customizing LMs in handling low-frequency entities on question answering tasks. We conduct extensive experiments on twelve LMs of varying size and type and different fine tuning, data augmentation, and retrieval models. Our findings indicate that while FT boosts the performance across entities of varying popularity, RAG surpasses FT by a large margin particularly for least popular factual knowledge. Additionally, the success of both RAG and FT approaches is amplified by improving retrieval and data augmentation techniques. Fine tuning, while beneficial for small LMs, requires extensive resources. To address this issue, we propose the new Stimulus RAG approach that surpasses the effectiveness of fine tuning based approaches, thereby eliminating the need for the costly data augmentation and fine tuning step for enriching LMs with less popular factual knowledge. The code is available at \url{https://github.com/informagi/RAGvsFT}.

• The paper demonstrates that while fine-tuning improves LLM performance on rare entities, retrieval augmented generation generally delivers superior results.
• The paper highlights that parameter-efficient fine-tuning methods like QLoRA offer modest gains but work effectively when combined with RAG.
• The paper shows that high-quality synthetic data and precise retrieval systems are crucial for optimal performance, especially in smaller models.

Analyzing Fine-Tuning Versus Retrieval Augmented Generation for Handling Low-Frequency Knowledge in LLMs

LLMs have demonstrated notable success across a broad range of tasks due to their capacity to memorize vast quantities of factual information. Nonetheless, their performance can decline when dealing with low-frequency or domain-specific entities. Two key approaches to enhance model performance in these contexts are Retrieval Augmented Generation (RAG) and Fine-Tuning (FT). This paper scrutinizes the impact these methods have on improving LLMs when confronted with low-frequency entities during open-domain question answering tasks.

Summary of Findings

The research presented indicates that fine-tuning significantly enhances performance, especially for entities within the extremes of popularity, although RAG consistently outperforms other methods. The efficacy of both strategies increases with advancements in retrieval and data augmentation techniques. Specifically, the paper's key findings are:

• Effective Strategies: RAG was shown to consistently outperform FT, particularly when used alongside fine-tuning. This synergy, however, dissipates in larger models due to enhanced internal memory capabilities.
• Fine-Tuning Variants: PEFT methods like QLoRA offer smaller performance improvements compared to full FT. However, when combined with RAG, PEFT methods prove beneficial, highlighting their ability to maintain inherent LLM reasoning capabilities.
• Synthetic Data Quality: Rather than the sheer volume, the quality of synthetic data profoundly impacts performance. Prompt-based data generation methods, for example, yielded stronger results compared to the end-to-end generation approach.
• Model Size and Retrieval Techniques: Larger models, with improved memorization capabilities, show reduced need for FT and RAG strategies for less popular knowledge. Nonetheless, the performance of both RAG and FT is closely tied to the retrieval system’s accuracy.

Practical and Theoretical Implications

Practically, the findings underscore the significance of tailoring approaches based on model size and the specific type of knowledge being dealt with. Industries deploying LLMs in specialized domains may consider adopting hybrid strategies that capitalize on both RAG and FT, especially when working with smaller models.

Theoretically, the paper advances our understanding of how retrieval and fine-tuning intersect to improve model performance with infrequent knowledge. It highlights the importance of the quality of both synthetic data and retrieval, shifting focus from merely expanding data volume. This understanding points towards the potential development of even more specialized tuning techniques or hybrid models that can dynamically adapt based on the type of query or context.

Future Directions

Future research could explore the application of these methodologies to more complex QA tasks, such as multi-hop and conversational QA. Further investigation into the development of advanced QA generation techniques could improve the quality of synthetic data, potentially enabling more cost-effective and efficient fine-tuning.

By offering insights into the nuanced impacts of RAG and FT, this paper contributes to the ongoing dialogue regarding the optimization of LLMs for domain-specific applications, potentially guiding future advances in the field of AI customization techniques.