Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought
(2210.01240)Abstract
LLMs have shown remarkable reasoning capabilities given chain-of-thought prompts (examples with intermediate reasoning steps). Existing benchmarks measure reasoning ability indirectly, by evaluating accuracy on downstream tasks such as mathematical reasoning. However, it is unclear how these models obtain the answers and whether they rely on simple heuristics rather than the generated chain-of-thought. To enable systematic exploration of the reasoning ability of LLMs, we present a new synthetic question-answering dataset called PrOntoQA, where each example is generated from a synthetic world model represented in first-order logic. This allows us to parse the generated chain-of-thought into symbolic proofs for formal analysis. Our analysis on InstructGPT and GPT-3 shows that LLMs are quite capable of making correct individual deduction steps, and so are generally capable of reasoning, even in fictional contexts. However, they have difficulty with proof planning: When multiple valid deduction steps are available, they are not able to systematically explore the different options.
Overview
-
Large language models like INSTRUCTGPT and GPT-3 show improved reasoning with chain-of-thought prompts, where intermediate steps are given.
-
A new synthetic dataset with first-order logic is created to better analyze LLMs' reasoning by turning their responses into formal proofs.
-
LLMs perform individual deduction steps well but struggle with proof planning and selecting between multiple valid deductive paths.
-
Performance is better when models use ontologies with real-world knowledge, indicating pre-training knowledge significantly aids reasoning.
-
Future research may explore advanced training strategies for LLMs to improve sophisticated reasoning, using methods like neuro-symbolic approaches.
Understanding LLMs and Reasoning
Overview of Chain-of-Thought Prompting
LLMs, such as INSTRUCTGPT and GPT-3, have demonstrated a significant capacity for complex reasoning when provided with chain-of-thought (CoT) prompts. CoT prompting is a technique where models are presented with examples containing intermediate reasoning steps, which has led to a marked improvement in answering logical questions. Nonetheless, understanding the mechanisms behind these models' reasoning processes, and whether they genuinely comprehend the reasoning steps or rely on heuristics, has remained a topic of investigation.
A New Synthetic Dataset for Analysis
Researchers at New York University have developed a synthetic dataset to delve into the reasoning capabilities of LLMs by examining the chains-of-thought generated by these models, rather than just focusing on the final answers. This dataset, formulated in first-order logic, enables the parsing of model-generated reasoning into formal proofs. Utilizing the dataset, an analysis of the performance of INSTRUCTGPT and GPT-3 has revealed that while these models manage individual deduction steps effectively, they struggle with proof planning, especially when faced with multiple valid deductive paths.
Insights into LLMs' Reasoning Abilities
The study shows that models exhibit competencies in reasoning even within fictional contexts. However, when it comes time to select among various valid proof steps or when tasked with extended proof sequences, the models demonstrate limitations. An interesting observation from the research is that models perform better with ontologies reflecting real-world knowledge. This finding implies that the substantial information LLMs acquire during their pre-training significantly influences their capacity for reasoning.
Conclusion and Implications
In summary, the research presents intriguing findings on the reasoning abilities of LLMs, highlighting their strengths and pinpointing areas where improvement is needed. Importantly, it points out the need for more advanced strategies in proof planning, potentially through methods like neuro-symbolic approaches. The findings set the stage for further exploration of how LLMs can be trained or fine-tuned in the future to enhance their capacity for sophisticated reasoning, beyond the present scope centered on modus ponens and shorter lengths of proofs.