Automated Design of Agentic Systems (2408.08435v2)

Abstract: Researchers are investing substantial effort in developing powerful general-purpose agents, wherein Foundation Models are used as modules within agentic systems (e.g. Chain-of-Thought, Self-Reflection, Toolformer). However, the history of machine learning teaches us that hand-designed solutions are eventually replaced by learned solutions. We describe a newly forming research area, Automated Design of Agentic Systems (ADAS), which aims to automatically create powerful agentic system designs, including inventing novel building blocks and/or combining them in new ways. We further demonstrate that there is an unexplored yet promising approach within ADAS where agents can be defined in code and new agents can be automatically discovered by a meta agent programming ever better ones in code. Given that programming languages are Turing Complete, this approach theoretically enables the learning of any possible agentic system: including novel prompts, tool use, workflows, and combinations thereof. We present a simple yet effective algorithm named Meta Agent Search to demonstrate this idea, where a meta agent iteratively programs interesting new agents based on an ever-growing archive of previous discoveries. Through extensive experiments across multiple domains including coding, science, and math, we show that our algorithm can progressively invent agents with novel designs that greatly outperform state-of-the-art hand-designed agents. Importantly, we consistently observe the surprising result that agents invented by Meta Agent Search maintain superior performance even when transferred across domains and models, demonstrating their robustness and generality. Provided we develop it safely, our work illustrates the potential of an exciting new research direction toward automatically designing ever-more powerful agentic systems to benefit humanity.

• The paper introduces ADAS, an automated framework that leverages meta agent search to design agentic systems without human-crafted prompts.
• Experiments show that automatically created agents improve performance by up to 14% across tasks like coding, reading comprehension, and mathematical problem-solving.
• The method reduces manual design efforts while ensuring cross-domain robustness, opening new avenues for scalable and innovative meta-learning in AI.

Automated Design of Agentic Systems

The paper "Automated Design of Agentic Systems" introduces and explores a nascent research area, namely Automated Design of Agentic Systems (ADAS). This field aims at the automated creation of agent systems by leveraging Foundation Models (FMs). This approach is posited as an alternative to manually designed agentic solutions, thus advancing the state of artificial intelligence through automation rather than human intervention.

Introduction and Core Idea

Central to this paper is the critique of the current paradigms in agentic system design. Traditional methods heavily rely on human expertise to craft systems using well-established techniques like Chain-of-Thought (COT), Self-Reflection, and various prompt engineering approaches. The authors argue that while these methods have demonstrated considerable merit, history in the machine learning domain suggests that hand-designed solutions often get supplanted by automated, learned approaches.

The authors propose ADAS to capitalize on this trend. By automating the design process, ADAS can invent novel building blocks and form complex agentic systems more efficiently than manual efforts. The theoretical foundation for this lies in the Turing completeness of programming languages, implying that any conceivable agentic system—including control flows, prompts, and tool integrations—can be discovered in code space.

Meta Agent Search Algorithm

To operationalize ADAS, the authors present Meta Agent Search, an algorithm where a meta agent iteratively programs new agentic systems. This process builds upon an archive of prior agents, continuously refining them. The framework built for Meta Agent Search includes basic essential functions like querying FMs and formatting prompts, ensuring the meta agent can create functional new agents efficiently.

The effectiveness of Meta Agent Search is demonstrated through several experiments across diverse domains including coding, science, and mathematics. The results are impressive, with the automatically generated agents significantly outperforming state-of-the-art hand-designed baselines.

Experimental Results

The quantitative results provide robust support for the efficiency of ADAS. Across multiple domains, the discovered agents exhibit superior performance:

1. ARC Challenge: The best-discovered agent outperforms established baselines like COT and Self-Consistency (COT-SC), achieving higher accuracy rates by up to 14%.
2. Reading Comprehension (DROP): The discovered agents demonstrate an increase in F1 scores by 13.6/100 compared to state-of-the-art agents.
3. Math (MGSM): The best agents show a remarkable improvement in accuracy by 14.4% over existing baselines.
4. Cross-Domain Transferability: The discovered agents maintain high performance even when transferred across different models and domains.

These empirical results indicate that Meta Agent Search not only finds superior solutions within specific tasks but also identifies robust general principles applicable across diverse settings.

Implications and Future Directions

From a practical perspective, the adoption of ADAS could drastically reduce the human labor involved in developing sophisticated agent systems. Given the scalability and efficiency of automated processes, ADAS can facilitate the rapid development and deployment of high-performance agents across a wide array of applications.

From a theoretical standpoint, ADAS opens up intriguing avenues of research in meta-learning and automated system design. Potential future developments include:

• Higher-order Meta-Learning: Investigating recursive applications of ADAS where the meta agent itself undergoes iterative improvements.
• Integration with Existing Frameworks: Leveraging existing agent frameworks and toolkits to seed ADAS, thereby building more complex and capable agents.
• Multi-objective Optimization: Incorporating objectives beyond mere performance, such as cost and latency, within ADAS algorithms.
• Exploration-Exploitation Trade-offs: Deploying advanced search algorithms to balance the need for discovering novel agents with the exploitation of already discovered high-performing systems.
• Safety and Robustness: Ensuring that agentic systems developed through ADAS are robust, interpretable, and safe for diverse, real-world applications.

Conclusion

The introduction of ADAS as an automated method for agentic system design marks a significant advancement in AI research. By shifting the focus from manual to automated design, this paper lays the groundwork for future innovations that could redefine the landscape of intelligent systems. As we continue to refine these technologies, the potential benefits of ADAS—to both the scientific community and society at large—are substantial. The continuous progress in this field promises to enhance both the efficiency and capability of next-generation AI agents, ultimately bridging gaps across domains and pushing the frontier of what is possible with artificial intelligence.