ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent (2312.10003v1)

Abstract: Answering complex natural language questions often necessitates multi-step reasoning and integrating external information. Several systems have combined knowledge retrieval with a LLM to answer such questions. These systems, however, suffer from various failure cases, and we cannot directly train them end-to-end to fix such failures, as interaction with external knowledge is non-differentiable. To address these deficiencies, we define a ReAct-style LLM agent with the ability to reason and act upon external knowledge. We further refine the agent through a ReST-like method that iteratively trains on previous trajectories, employing growing-batch reinforcement learning with AI feedback for continuous self-improvement and self-distillation. Starting from a prompted large model and after just two iterations of the algorithm, we can produce a fine-tuned small model that achieves comparable performance on challenging compositional question-answering benchmarks with two orders of magnitude fewer parameters.

• The paper demonstrates a self-improving multi-step reasoning agent that integrates ReAct and ReST to iteratively enhance LLM performance without human labels.
• It employs a search loop, AI feedback, and few-shot Python prompts to refine responses, achieving accuracy gains with smaller models.
• Experimental results indicate significant improvements after two iterations, matching performance of larger pre-trained models.

ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent

Introduction

The paper discusses a strategy to enhance the performance of LLM (LM) agents, specifically for complex natural language tasks requiring multi-step reasoning. It integrates ReAct and ReST approaches to create a self-improving multi-step reasoning agent, enabling it to refine its process through AI feedback, without human-labeled training data. The approach utilizes a combination of retrieval and generation techniques for question-answering tasks, aiming to optimize the agent's ability to interact with external knowledge.

ReAct and ReST Approaches

The ReAct framework allows for reasoning intertwined with execution actions, forming a foundation for the proposed agent. ReACT agents systematically intersperse thought, action, and observation cycles to construct answers based on information retrieval. On the other hand, ReST focuses on iterative training using previously gathered data, akin to reinforcement learning methods. Here, AI feedback facilitates the refinement of processes instead of direct outcomes, thus propelling the agent towards self-improvement.

System Architecture

The architecture of the proposed system involves multiple decision layers, including a search loop and subsequent self-revision steps. The agent begins by exploring whether additional information is needed for a given question, employing search queries as necessary. Each query retrieves relevant snippets, which are summarized and contribute to generating a preliminary response. The response is then verified for relevance and grounding, refining it further before producing the final answer.

The agent relies on pre-designed, few-shot prompts formatted in Python code to guide reasoning tasks, ensuring outputs are structured and integration-ready. This format further aids in models' understanding and predictions, particularly when a structural and descriptive format like Python is employed.

Experimental Setup

The experiments were conducted using PaLM 2 models of various sizes (XS, S, and L), with the agent evaluated on datasets like Bamboogle and BamTwoogle. These datasets contain compositional, multi-hop reasoning questions unanswerable through direct search queries alone. The evaluation employed auto-evaluation mechanisms aligned with human judgments, utilizing a distinct dataset to maintain training and evaluation separation.

Results

The experiments revealed that the proposed self-improvement strategy significantly enhances model performance. After two iterations, performance comparable to much larger pre-trained models was achieved with a substantially smaller model. This illustrates the efficacy of process-driven, feedback-based training in resource-constrained setups. Both auto-evaluation and human evaluation confirmed these gains, with notable improvements in accuracy over successive iterations.

Applying self-critique mechanisms demonstrated a minor boost in performance, highlighting the utility of iterative validation within multi-step reasoning contexts. However, fine-tuning on human-filtered data showed marginal benefits, suggesting the algorithm's robustness against noisy data.

Implications and Future Work

The methodology has practical implications for scalable AI systems, where human-labeling is infeasible or costly. By leveraging synthetic data for iterative model refinement, LLM agents can progressively improve without extensive labeled datasets. Future research might expand this framework to incorporate more diverse toolsets and examine the scalability of iterative improvements with additional model generations.

Conclusion

This work presents a feasible and effective approach to enhancing LLM agents' reasoning capabilities via self-improvement. By integrating ReAct and ReST methodologies under a structured, feedback-oriented training regime, the research illustrates a path toward more autonomous, efficient, and scalable LLMs capable of complex multi-step reasoning. This model paves the way for models demanding less human oversight while simultaneously improving their interpretative and interactive competences.