Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (2212.10509v2)

Published 20 Dec 2022 in cs.CL

Abstract: Prompting-based LLMs are surprisingly powerful at generating natural language reasoning steps or Chains-of-Thoughts (CoT) for multi-step question answering (QA). They struggle, however, when the necessary knowledge is either unavailable to the LLM or not up-to-date within its parameters. While using the question to retrieve relevant text from an external knowledge source helps LLMs, we observe that this one-step retrieve-and-read approach is insufficient for multi-step QA. Here, \textit{what to retrieve} depends on \textit{what has already been derived}, which in turn may depend on \textit{what was previously retrieved}. To address this, we propose IRCoT, a new approach for multi-step QA that interleaves retrieval with steps (sentences) in a CoT, guiding the retrieval with CoT and in turn using retrieved results to improve CoT. Using IRCoT with GPT3 substantially improves retrieval (up to 21 points) as well as downstream QA (up to 15 points) on four datasets: HotpotQA, 2WikiMultihopQA, MuSiQue, and IIRC. We observe similar substantial gains in out-of-distribution (OOD) settings as well as with much smaller models such as Flan-T5-large without additional training. IRCoT reduces model hallucination, resulting in factually more accurate CoT reasoning. Code, data, and prompts are available at \url{https://github.com/stonybrooknlp/ircot}

Definition Search Book Streamline Icon: https://streamlinehq.com
References (36)
  1. Learning to retrieve reasoning paths over wikipedia graph for question answering. In International Conference on Learning Representations.
  2. Improving language models by retrieving from trillions of tokens. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 2206–2240. PMLR.
  3. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  4. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  5. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416.
  6. Multi-step retriever-reader interaction for scalable open-domain question answering. In International Conference on Learning Representations.
  7. Yair Feldman and Ran El-Yaniv. 2019. Multi-hop paragraph retrieval for open-domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2296–2309, Florence, Italy. Association for Computational Linguistics.
  8. IIRC: A dataset of incomplete information reading comprehension questions. In EMNLP.
  9. Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3929–3938. PMLR.
  10. Large language models are reasoning teachers. arXiv preprint arXiv:2212.10071.
  11. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In COLING.
  12. Atlas: Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299.
  13. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics.
  14. RealTime QA: What’s the answer right now? arXiv preprint arXiv:2207.13332.
  15. Baleen: Robust multi-hop reasoning at scale via condensed retrieval. In Advances in Neural Information Processing Systems.
  16. Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive NLP.
  17. Decomposed prompting: A modular approach for solving complex tasks.
  18. Decomposed prompting: A modular approach for solving complex tasks. In The Eleventh International Conference on Learning Representations.
  19. Large language models are zero-shot reasoners. In ICML 2022 Workshop on Knowledge Retrieval and Language Models.
  20. Internet-augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115.
  21. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474. Curran Associates, Inc.
  22. Teaching small language models to reason. arXiv preprint arXiv:2212.08410.
  23. WebGPT: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332.
  24. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems.
  25. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350.
  26. Answering complex open-domain questions through iterative query generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2590–2602, Hong Kong, China. Association for Computational Linguistics.
  27. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389.
  28. Recitation-augmented language models. arXiv preprint arXiv:2210.01296.
  29. UL2: Unifying language learning paradigms. In The Eleventh International Conference on Learning Representations.
  30. MuSiQue: Multihop questions via single-hop question composition. TACL, 10:539–554.
  31. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
  32. Answering complex open-domain questions with multi-hop dense retrieval. In International Conference on Learning Representations.
  33. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP.
  34. ReAct: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629.
  35. Generate rather than retrieve: Large language models are strong context generators. In The Eleventh International Conference on Learning Representations.
  36. Retrieving and reading: A comprehensive survey on open-domain question answering. arXiv preprint arXiv:2101.00774.
Citations (277)
View on Semantic Scholar

Summary

  • The paper introduces IRCoT, a method that interlaces retrieval with chain-of-thought reasoning to incrementally gather evidence and improve multi-step QA.
  • The paper demonstrates IRCoT's effectiveness across benchmarks like HotpotQA and 2WikiMultihopQA, significantly outperforming single-step retrieval baselines.
  • The paper highlights IRCoT's robustness with various LLM sizes and out-of-distribution data, ensuring enhanced recall and factual accuracy.

Introduction

The paradigm of prompting LLMs to perform natural language reasoning in a step-by-step fashion, recognized as chains of thoughts (CoT), has seen significant success, especially for tasks where all necessary information is assumed to appear in the LLM's learned parameters. However, LLMs often falter when external, up-to-the-minute knowledge is required for open-domain multi-step question answering (QA). Traditional solutions to augment LLMs have involved a singular retrieval step, but this is less effective for complex queries where retrieved information is needed incrementally, as reasoning progresses.

Interleaved Retrieval and CoT (IRCoT) Method

Proposed in this work is a method called Interleaved Retrieval guided by Chain-of-Thought (IRCoT), an approach that interlaces retrieval with the CoT process. Initially, documents are extracted using the question as a query. During the answering process, retrieval and reasoning mutually inform each subsequent action. The CoT generation step builds on existing reasoning and collected paragraphs to craft the next sentence of the CoT sequence. Reversely, the newly generated CoT sentence guides the retrieval of additional evidence. This cycle repeats until a termination criterion is reached, contemporaneously enhancing both CoT generation quality and relevance of retrieved information.

Efficacy of IRCoT

IRCoT's performance has been evaluated across multiple datasets: HotpotQA, 2WikiMultihopQA, MuSiQue, and IIRC. Employing GPT3 and Flan-T5 models, the IRCoT approach surpasses baseline single-step retrieval by a substantial margin, both in terms of recall and QA performance. Furthermore, IRCoT demonstrates robustness in out-of-distribution scenarios and is effective even with smaller LLMs. To aid practical replication and future research, resources including code and data prompts are publicly accessible online.

Conclusions and Further Remarks

IRCoT stands as a notable approach, intertwining retrieval and CoT generation to navigate open-domain, multi-step QA tasks effectively. This technique elevates both the accuracy of retrieved information and the factual reliability of generated CoTs, with gains witnessed across diverse LLM sizes and testing conditions. Despite its certain dependencies on specific LLM capabilities, such as zero or few-shot CoT generation ability and support for longer contexts, IRCoT represents a stride forward in the domain of knowledge-intensive QA, potentially informing a variety of future LLM applications.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube