Generate then Retrieve: Conversational Response Retrieval Using LLMs as Answer and Query Generators
(2403.19302)Abstract
CIS is a prominent area in IR that focuses on developing interactive knowledge assistants. These systems must adeptly comprehend the user's information requirements within the conversational context and retrieve the relevant information. To this aim, the existing approaches model the user's information needs with one query called rewritten query and use this query for passage retrieval. In this paper, we propose three different methods for generating multiple queries to enhance the retrieval. In these methods, we leverage the capabilities of LLMs in understanding the user's information need and generating an appropriate response, to generate multiple queries. We implement and evaluate the proposed models utilizing various LLMs including GPT-4 and Llama-2 chat in zero-shot and few-shot settings. In addition, we propose a new benchmark for TREC iKAT based on gpt 3.5 judgments. Our experiments reveal the effectiveness of our proposed models on the TREC iKAT dataset.
Overview
-
The paper proposes methods to enhance conversational response retrieval using LLMs, aiming to address the limitations of existing systems that fail to capture complex information needs.
-
Three novel approaches are introduced: Answer-driven Query Generation (AD), Query Generation (QD), and Answer and Query Generation (AQD), with an additional variant AQDAnswer that re-ranks results for better passage retrieval.
-
Experiments on the TREC iKAT dataset show that the AQD and AD methods, especially with GPT-4, significantly outperform baselines, indicating the benefit of multiple queries and re-ranking strategies.
-
The study suggests the importance of leveraging LLMs not just for response generation but within the retrieval process itself, opening avenues for future work on optimizing query generation and integrating user feedback.
Generate then Retrieve: Enhancing Conversational Response Retrieval with LLMs
Methods Overview
The paper introduces novel approaches to improve conversational response retrieval by leveraging LLMs. It identifies the main limitation of existing retrieval systems, which typically employ a single rewritten query for passage retrieval, failing to address complex information needs that require reasoning over multiple facts. To overcome this, the authors propose three methods:
- Answer-driven Query Generation (AD): Utilizing the LLM's generated answer as a singular long query for retrieval.
- Query Generation (QD): Prompting the LLM to directly generate multiple queries from the conversational context.
- Answer and Query Generation (AQD): A two-step method where the LLM first generates an answer and then produces multiple queries to refine this answer.
An additional variant, AQDAnswer, re-ranks results based on predicted relevance to the LLM's generated response, aiming to improve the quality of retrieved passages. The paper compares these methods against standard approaches and evaluates them using LLMs including GPT-4 and Llama-2 in different settings.
Experimental Setup and Results
The experiments are conducted on the TREC Interactive Knowledge Assistance Track (iKAT) dataset, showcasing the complexity of conversational information seeking tasks. The proposed methods are evaluated against baselines that follow either generate-then-retrieval or retrieval-then-generate paradigms, using a variety of LLMs.
Results indicate that AQD and AD methods, particularly when utilizing GPT-4, significantly outperform the baselines. AQD shows superior performance over single-query rewriting approaches (QR) and even outpaces human-rewritten queries in certain metrics. Notably, AQDAnswer's re-ranking strategy based on the initial generated answer leads to further improvements, showcasing the potential of LLMs in enhancing retrieval through a nuanced understanding of the conversational context and the user's information need.
Implications and Future Work
This study presents a significant shift towards utilizing the generative capabilities of LLMs for improving information retrieval in conversational systems. By demonstrating that multiple queries generated from LLMs' responses can lead to better retrieval outcomes, it opens up new avenues for research in conversational search systems. It also highlights the importance of leveraging LLMs not just for generating responses but as integral components of the information retrieval process.
One promising direction for future work is exploring the optimal number of queries to generate and the impact of query quality on retrieval effectiveness. Additionally, integrating user feedback into the generative process could further personalize and refine the retrieval outcomes, making the conversational system more responsive to the user's specific needs.
Ethical Considerations and Limitations
The reliance on LLMs introduces potential biases and errors inherent in these models, which can affect the quality of generated responses and queries. Moreover, the effectiveness of the proposed methods is contingent upon the quality of the LLM's initial response, highlighting a dependency that could be problematic if the LLM fails to understand the user's request accurately. Future research should address these challenges, ensuring that conversational systems remain reliable, unbiased, and user-centric in their approach to information retrieval.