CLARINET: Augmenting Language Models to Ask Clarification Questions for Retrieval (2405.15784v1)
Abstract: Users often make ambiguous requests that require clarification. We study the problem of asking clarification questions in an information retrieval setting, where systems often face ambiguous search queries and it is challenging to turn the uncertainty in the retrieval model into a natural language question. We present CLARINET, a system that asks informative clarification questions by choosing questions whose answers would maximize certainty in the correct candidate. Our approach works by augmenting a LLM to condition on a retrieval distribution, finetuning end-to-end to generate the question that would have maximized the rank of the true candidate at each turn. When evaluated on a real-world retrieval dataset of users searching for books, our system outperforms traditional heuristics such as information gain on retrieval success by 17% and vanilla-prompted LLMs by 39% relative.
- ’it’s on the tip of my tongue’ a new dataset for known-item retrieval. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pages 48–56.
- Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416.
- Robert Hawkins and Noah Goodman. 2017. Why do you ask? the informational dynamics of questions and answers.
- Gautier Izacard and Edouard Grave. 2020. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282.
- Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906.
- Kimiya Keyvan and Jimmy Xiangji Huang. 2022. How to approach ambiguous queries in conversational search: A survey of techniques, approaches, tools, and challenges. ACM Computing Surveys, 55(6):1–40.
- Decomposing complex queries for tip-of-the-tongue retrieval. arXiv preprint arXiv:2305.15053.
- Experience matters: Information acquisition optimizes probability gain. Psychological science, 21(7):960–969.
- Mike Oaksford and Nick Chater. 1994. A rational analysis of the selection task as optimal data selection. Psychological review, 101(4):608.
- Reinforced clarification question generation with defeasibility rewards for disambiguating social and moral situations. arXiv preprint arXiv:2212.10409.
- Sudha Rao and Hal Daumé III. 2018. Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information. arXiv preprint arXiv:1805.04655.
- Sudha Rao and Hal Daumé III. 2019. Answer-based adversarial training for generating clarification questions. arXiv preprint arXiv:1904.02281.
- Question asking as program generation. Advances in neural information processing systems, 30.
- Robert Van Rooy. 2004. Utility, informativity and protocols. Journal of philosophical logic, 33:389–419.
- Open-domain clarification question generation without question examples. arXiv preprint arXiv:2110.09779.
- Interactive classification by asking informative questions. arXiv preprint arXiv:1911.03598.
- Generating clarifying questions for information retrieval. In Proceedings of the web conference 2020, pages 418–428.
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
