Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Reflexion: Language Agents with Verbal Reinforcement Learning (2303.11366v4)

Published 20 Mar 2023 in cs.AI, cs.CL, and cs.LG

Abstract: LLMs have been increasingly used to interact with external environments (e.g., games, compilers, APIs) as goal-driven agents. However, it remains challenging for these language agents to quickly and efficiently learn from trial-and-error as traditional reinforcement learning methods require extensive training samples and expensive model fine-tuning. We propose Reflexion, a novel framework to reinforce language agents not by updating weights, but instead through linguistic feedback. Concretely, Reflexion agents verbally reflect on task feedback signals, then maintain their own reflective text in an episodic memory buffer to induce better decision-making in subsequent trials. Reflexion is flexible enough to incorporate various types (scalar values or free-form language) and sources (external or internally simulated) of feedback signals, and obtains significant improvements over a baseline agent across diverse tasks (sequential decision-making, coding, language reasoning). For example, Reflexion achieves a 91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4 that achieves 80%. We also conduct ablation and analysis studies using different feedback signals, feedback incorporation methods, and agent types, and provide insights into how they affect performance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (31)
  1. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691.
  2. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
  3. In-context policy iteration. arXiv preprint arXiv:2210.03821.
  4. Multipl-e: A scalable and extensible approach to benchmarking neural code generation.
  5. Codet: Code generation with generated tests. arXiv preprint arXiv:2207.10397.
  6. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  7. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128.
  8. Textworld: A learning environment for text-based games. In Computer Games: 7th Workshop, CGW 2018, Held in Conjunction with the 27th International Conference on Artificial Intelligence, IJCAI 2018, Stockholm, Sweden, July 13, 2018, Revised Selected Papers 7, pages 41–75. Springer.
  9. Goodman, N. (2023). Meta-prompt: A simple self-improving language agent. noahgoodman.substack.com.
  10. Language models can solve computer tasks. arXiv preprint arXiv:2303.17491.
  11. A large-scale longitudinal study of flaky tests. Proc. ACM Program. Lang., 4(OOPSLA).
  12. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314–21328.
  13. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161.
  14. Competition-level code generation with alphacode. Science, 378(6624):1092–1097.
  15. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651.
  16. Dera: Enhancing large language model completions with dialog-enabled resolving agents. arXiv preprint arXiv:2303.17071.
  17. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332.
  18. OpenAI (2023). Gpt-4 technical report. ArXiv.
  19. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442.
  20. Refiner: Reasoning feedback on intermediate representations. arXiv preprint arXiv:2304.01904.
  21. Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint arXiv:2305.03495.
  22. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761.
  23. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580.
  24. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In Proceedings of the International Conference on Learning Representations (ICLR).
  25. Reinforcement Learning: An Introduction. The MIT Press, second edition.
  26. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903.
  27. Decomposition enhances reasoning via self-evaluation guided decoding. arXiv preprint arXiv:2305.00633.
  28. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP).
  29. Webshop: Towards scalable real-world web interaction with grounded language agents. In ArXiv.
  30. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR).
  31. Answering questions by meta-reasoning over multiple chains of thought. arXiv preprint arXiv:2304.13007.
Citations (772)
View on Semantic Scholar

Summary

  • The paper presents Reflexion, a paradigm that employs internal verbal feedback to enhance decision-making in language agents.
  • It utilizes an episodic memory to store self-generated reflective feedback, achieving a 91% pass@1 accuracy on code generation benchmarks.
  • The approach offers computational efficiency and interpretable action hints, paving the way for advances in autonomous reasoning and agent training.

Overview

The field of LLMs has witnessed a significant transformation with the introduction of Reflexion, a novel paradigm that shifts the focus towards enhancing language agents through verbal reinforcement. This method diverges from traditional reinforcement learning techniques, which predominantly rely on extensive training data and model fine-tuning, by employing linguistic feedback for agent improvement.

The Essence of Reflexion

Reflexion stands out by allowing agents to internally generate reflective textual feedback based on their performance in various tasks. This reflective feedback is then stored in an episodic memory, enabling the agent to make more informed decisions in future attempts. This process mirrors human learning patterns where reflection on past experiences leads to improved future actions. Remarkably, Reflexion is versatile, accommodating different types of feedback signals and sources, whether they are external or internally simulated.

Comparative Advantages

Traditional reinforcement learning (RL) methods, though effective, come with their set of challenges, including the need for substantial computational power and intricacies in performing accurate credit assignments with scalar or vector rewards. Reflexion addresses these challenges by:

  • Being computationally efficient as it doesn't necessitate fine-tuning of the LLM.
  • Offering a nuanced feedback system that transcends basic scalar or vector rewards, thus providing more targeted action adjustments.
  • Enabling a more explicit and interpretable episodic memory of prior experiences.
  • Furnishing more explicit action hints for future episodes.

Empirical Evidences

The effectiveness of Reflexion is underscored by its impressive performance across a spectrum of tasks including sequential decision-making, reasoning, and programming. Notably, it achieved a 91% pass@1 accuracy on the HumanEval coding benchmark, outperforming the previous state-of-the-art GPT-4, which secured an 80% accuracy. This stark improvement highlights Reflexion's potential to redefine benchmarks in generative AI tasks.

Experimental Insights

Reflexion's integration into tasks like the AlfWorld suite and HotPotQA showcased its ability to substantially boost agent performance by up to 22% and 20% respectively over traditional approaches. These experiments underline Reflexion’s proficiency in not only interpreting the task at hand but also in leveraging past experiences to enhance future task execution. In programming tasks, Reflexion not only set new benchmarks in code generation accuracy but also demonstrated its language-agnostic capability, offering promising implications for a wide range of programming languages.

Limitations and Future Directions

While Reflexion introduces a groundbreaking approach to enabling agents to learn from linguistic feedback, it's essential to acknowledge its limitations. The simplification of retaining episodic memory to a fixed size may not always encapsulate the depth of experiences needed for complex decision-making. Future work could explore the expansion of memory mechanisms and delve into more sophisticated models that encompass a broader spectrum of learning strategies, mirroring human cognitive processes more closely.

Conclusion

Reflexion represents a significant leap forward in the development of intelligent language agents, offering a novel and effective approach to learning through verbal reinforcement. By enabling agents to self-reflect and learn from their experiences, Reflexion poses to significantly advance the capabilities of generative AI, pushing the boundaries of what's possible in autonomous decision-making and reasoning tasks.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube