Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity
(2403.14403)Abstract
Retrieval-Augmented LLMs, which incorporate the non-parametric knowledge from external knowledge bases into LLMs, have emerged as a promising approach to enhancing response accuracy in several tasks, such as Question-Answering (QA). However, even though there are various approaches dealing with queries of different complexities, they either handle simple queries with unnecessary computational overhead or fail to adequately address complex multi-step queries; yet, not all user requests fall into only one of the simple or complex categories. In this work, we propose a novel adaptive QA framework, that can dynamically select the most suitable strategy for (retrieval-augmented) LLMs from the simplest to the most sophisticated ones based on the query complexity. Also, this selection process is operationalized with a classifier, which is a smaller LM trained to predict the complexity level of incoming queries with automatically collected labels, obtained from actual predicted outcomes of models and inherent inductive biases in datasets. This approach offers a balanced strategy, seamlessly adapting between the iterative and single-step retrieval-augmented LLMs, as well as the no-retrieval methods, in response to a range of query complexities. We validate our model on a set of open-domain QA datasets, covering multiple query complexities, and show that ours enhances the overall efficiency and accuracy of QA systems, compared to relevant baselines including the adaptive retrieval approaches. Code is available at: https://github.com/starsuzi/Adaptive-RAG.
Overview
-
Adaptive-RAG introduces a novel framework for Open-domain Question Answering that dynamically selects the best retrieval strategy based on query complexity, choosing between non-retrieval, one-step, and multi-step approaches.
-
A smaller LLM classifier is trained to predict query complexity using automatically annotated query-complexity pairs, guiding the selection of the most appropriate retrieval strategy.
-
Experimental results show Adaptive-RAG outperforms existing adaptive retrieval strategies on a mix of single-hop and multi-hop open-domain QA datasets, demonstrating improved accuracy and computational efficiency.
-
The framework opens up future research directions focused on enhancing query complexity classifiers and exploring Adaptive-RAG's applicability to broader NLP tasks.
Adaptive Retrieval-Augmented Generation for Open-Domain Question Answering
Introduction to Adaptive-RAG
Retrieval-Augmented Generation (RAG) models have marked a significant advancement in the Open-domain Question Answering (QA) domain by effectively incorporating external knowledge into LLMs. Among various strategies, the Single-step Approach and Multi-step Approach models represented two main paradigms addressing queries of varying complexities. However, neither approach universally sufficed—while the former might underperform on complex queries requiring multi-faceted reasoning, the latter could introduce unnecessary computational overhead for simpler queries. Addressing this gap, the novel Adaptive Retrieval-Augmented Generation (Adaptive-RAG) framework presents itself as a dynamic solution. It selects the most fitting RAG strategy—from relying solely on LLM, through single-step to multi-step retrieval—based on the assessed query complexity. This summary elucidates the framework’s methodology, validated across benchmark datasets, showcasing its efficacy in enhancing QA accuracy while ensuring computational efficiency.
Methodology
The Adaptive-RAG framework capitalizes on a classifier to pre-determine the complexity of an input query, subsequently selecting an appropriate retrieval strategy. Central strategies span across non-retrieval (A'), one-step retrieval (B'), and multi-step retrieval (`C') approaches, each tailored to different complexity levels.
The classifier is a smaller LLM trained on automatically annotated query-complexity pairs, leveraging the outcomes of differing RAG strategies and inductive biases existing within datasets. The labeling methodology inherently assumes that simple queries, correctly answered by less complex strategies or inherent in single-hop datasets, correspond to lower complexity levels, whereas queries demanding multi-hop reasoning are classified at higher complexity levels.
Experimental Results
Validated on a selection of open-domain QA datasets, incorporating both single-hop and multi-hop queries, the Adaptive-RAG demonstrated a balanced trade-off between efficiency and accuracy. Specifically, it outperformed existing adaptive retrieval strategies, signaling a noteworthy advancement in adaptively addressing queries of diverse complexities.
Besides, the adaptive framework underscored the significance of an effective query complexity classifier. It highlighted that precise classification facilitates optimal strategy selection, thereby improving QA performance and resource utilization. Moreover, comparative analysis across different LLMs and model sizes reinforced Adaptive-RAG’s robustness and scalability.
Theoretical and Practical Implications
Adaptive-RAG introduces a nuanced perspective on handling query complexity in RAG frameworks, pivoting from a one-size-fits-all approach to a dynamic, complexity-aware strategy selection. Theoretically, it underscores the essentiality of accurately assessing query complexity and the potential of adaptive systems in significantly enhancing model performance without necessitating architectural adjustments.
Practically, Adaptive-RAG promises substantial efficiency improvements, especially critical given the computational demands of contemporary LLMs. By effectively balancing resource allocation, it presents a viable avenue for scalable real-world applications, offering a pathway towards more responsive and accurate QA systems.
Future Directions
While Adaptive-RAG marks a significant stride in the right direction, the development of more sophisticated query complexity classifiers emerges as a promising avenue for future research. Enhancements in classifier accuracy are anticipated to further amplify the framework’s efficiency and efficacy. Additionally, exploring Adaptive-RAG’s applicability across broader NLP tasks may unveil its versatile potential in enhancing LLM-based applications.
In summary, Adaptive-RAG embodies a significant leap toward more intelligent, adaptable, and efficient retrieval-augmented systems, setting a precedent for future advancements in the domain of open-domain question answering.
Create an account to read this summary for free: