LLM Guided Inductive Inference for Solving Compositional Problems (2309.11688v1)
Abstract: While LLMs have demonstrated impressive performance in question-answering tasks, their performance is limited when the questions require knowledge that is not included in the model's training data and can only be acquired through direct observation or interaction with the real world. Existing methods decompose reasoning tasks through the use of modules invoked sequentially, limiting their ability to answer deep reasoning tasks. We introduce a method, Recursion based extensible LLM (REBEL), which handles open-world, deep reasoning tasks by employing automated reasoning techniques like dynamic planning and forward-chaining strategies. REBEL allows LLMs to reason via recursive problem decomposition and utilization of external tools. The tools that REBEL uses are specified only by natural language description. We further demonstrate REBEL capabilities on a set of problems that require a deeply nested use of external tools in a compositional and conversational setting.
- Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
- Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
- Compositional semantic parsing with large language models. arXiv preprint arXiv:2209.15003, 2022.
- Towards reasoning in large language models: A survey. arXiv preprint arXiv:2212.10403, 2022.
- Measuring compositional generalization: A comprehensive method on realistic data. arXiv preprint arXiv:1912.09713, 2019.
- Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406, 2022.
- Koroteev, M. Bert: a review of applications in natural language processing and understanding. arXiv preprint arXiv:2103.11943, 2021.
- Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In International conference on machine learning, pp. 2873–2882. PMLR, 2018.
- Liebowitz, J. Introduction to expert systems. Mitchell Publishing, Inc., 1988.
- Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35, 2023.
- Chameleon: Plug-and-play compositional reasoning with large language models. arXiv preprint arXiv:2304.09842, 2023.
- Language models are few-shot butlers. arXiv preprint arXiv:2104.07972, 2021.
- Neural unification for logic reasoning over natural language. arXiv preprint arXiv:2109.08460, 2021.
- Measuring and narrowing the compositionality gap in language models. arXiv:2210.03350v1, 2022.
- Reasoning with language model prompting: A survey. arXiv preprint arXiv:2212.09597, 2022.
- Russell, S. J. Artificial intelligence a modern approach. Pearson Education, Inc., 2010.
- Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023.
- Large language model programs. arXiv:2305.05364, 2023.
- Fever: a large-scale dataset for fact extraction and verification. arXiv preprint arXiv:1803.05355, 2018.
- Chain-of-thought prompting elicits reasoning in large language models. NeurIPS, 2022.
- Seqzero: Few-shot compositional semantic parsing with sequential prompts and zero-shot models. arXiv preprint arXiv:2205.07381, 2022.
- Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600, 2018.
- React: Synergizing reasoning and acting in language models. ICLR, 2023.
- Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625, 2022.
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.