Abstract
LLMs inevitably exhibit hallucinations since the accuracy of generated texts cannot be secured solely by the parametric knowledge they encapsulate. Although retrieval-augmented generation (RAG) is a practicable complement to LLMs, it relies heavily on the relevance of retrieved documents, raising concerns about how the model behaves if retrieval goes wrong. To this end, we propose the Corrective Retrieval Augmented Generation (CRAG) to improve the robustness of generation. Specifically, a lightweight retrieval evaluator is designed to assess the overall quality of retrieved documents for a query, returning a confidence degree based on which different knowledge retrieval actions can be triggered. Since retrieval from static and limited corpora can only return sub-optimal documents, large-scale web searches are utilized as an extension for augmenting the retrieval results. Besides, a decompose-then-recompose algorithm is designed for retrieved documents to selectively focus on key information and filter out irrelevant information in them. CRAG is plug-and-play and can be seamlessly coupled with various RAG-based approaches. Experiments on four datasets covering short- and long-form generation tasks show that CRAG can significantly improve the performance of RAG-based approaches.
Overview
-
Large language models suffer from factual errors and hallucinations, which Retrieval-Augmented Generation systems try to mitigate by using external documents.
-
CRAG introduces a retrieval evaluator that categorizes document relevance into Correct, Incorrect, or Ambiguous to improve RAG robustness by processing only pertinent information.
-
CRAG integrates with web searches to access a broader range of information when existing corpus knowledge is inadequate, demonstrating its plug-and-play nature and compatibility with RAG methods.
-
Empirical validation of CRAG over four datasets highlights enhancements in both short and long-form generation tasks, outperforming standard RAG and Self-RAG approaches.
-
The paper emphasizes CRAG's contribution to robust generation by refining information retrieval and use, illustrating significant performance improvements and generalizability.
Introduction
LLMs have gained significant traction for their impressive language generation abilities. However, they often grapple with factual errors and hallucinations, highlighting inherent limitations in their parametric knowledge. While Retrieval-Augmented Generation (RAG) systems offer a practical supplement to LLMs by augmenting generation with external documents, their efficacy is critically dependent on the relevance and accuracy of these documents. Current approaches, however, may indiscriminately integrate irrelevant information, undermining the robustness of generation.
Corrective Strategies for Robust Generation
The paper introduces an innovative Corrective Retrieval Augmented Generation (CRAG) methodology designed to enhance the resilience of RAG systems. CRAG incorporates a lightweight retrieval evaluator alongside a varying approach to document use depending on the evaluator's output. The evaluator assigns actions based on the estimated relevance, falling under Correct, Incorrect, or Ambiguous categories. This approach pivots on a decomposition-recomposition algorithm which addresses static corpuses' limitations by selectively focusing on essential information and discarding irrelevant content.
Web Search Integration and Experimentation
A novel aspect of CRAG is its integration with large-scale web searches for cases when knowledge retrieval is deemed Incorrect. Leveraging the dynamic nature of the web, CRAG broadens the scope and variety of information at the model’s disposal. This ensures a rich set of external knowledge to amend the initial corpus results. CRAG is demonstrated to be a plug-and-play model and shows compatibility with existing RAG-based approaches. Validation across four datasets emphasizes its performance uplift and generalizability over tasks that demand both short and long-form generation.
Conclusion and Contribution
CRAG's design for self-correcting and enhancing the utilization of retrieved documents marks a notable step in addressing RAG's existing pitfalls. The retrieval evaluator is central, helping to avoid the inclusion of misleading information and prompting robust actions based on document assessment. The refined approach shows significant improvements over the standard RAG and Self-RAG, indicating wide applicability in scenarios where the integrity of retrieved information is questionable. The experiments encapsulate CRAG's adaptability to RAG-based methods and its capability to transcend challenges across various generation categories.