Alibaba LingmaAgent: Improving Automated Issue Resolution via Comprehensive Repository Exploration (2406.01422v2)

Abstract: This paper presents Alibaba LingmaAgent, a novel Automated Software Engineering method designed to comprehensively understand and utilize whole software repositories for issue resolution. Deployed in TONGYI Lingma, an IDE-based coding assistant developed by Alibaba Cloud, LingmaAgent addresses the limitations of existing LLM-based agents that primarily focus on local code information. Our approach introduces a top-down method to condense critical repository information into a knowledge graph, reducing complexity, and employs a Monte Carlo tree search based strategy enabling agents to explore and understand entire repositories. We guide agents to summarize, analyze, and plan using repository-level knowledge, allowing them to dynamically acquire information and generate patches for real-world GitHub issues. In extensive experiments, LingmaAgent demonstrated significant improvements, achieving an 18.5\% relative improvement on the SWE-bench Lite benchmark compared to SWE-agent. In production deployment and evaluation at Alibaba Cloud, LingmaAgent automatically resolved 16.9\% of in-house issues faced by development engineers, and solved 43.3\% of problems after manual intervention. Additionally, we have open-sourced a Python prototype of LingmaAgent for reference by other industrial developers https://github.com/RepoUnderstander/RepoUnderstander. In fact, LingmaAgent has been used as a developed reference by many subsequently agents.

• The paper proposes RepoUnderstander, an agent-based method that builds hierarchical repository graphs and uses MCTS to capture global code interdependencies.
• It addresses the limitations of local code analysis by integrating comprehensive knowledge of repository structure for tasks like fault localization and program repair.
• Empirical results show an 18.5% improvement and a 21.33% problem-solving rate, highlighting its practical impact in automatic software engineering.

Analyzing "How to Understand Whole Software Repository?"

The paper "How to Understand Whole Software Repository?" authored by Yingwei Ma, Qingping Yang, Rongyu Cao, Binhua Li, Fei Huang, and Yongbin Li from Alibaba Group, proposes a significant advancement in the field of Automatic Software Engineering (ASE). This work introduces RepoUnderstander, an agent-based method designed to guide LLM-based agents to acquire a comprehensive understanding of entire software repositories.

Core Contributions

1. Problem Context

The authors situate their work within the broader scope of ASE, acknowledging recent advancements driven by LLM-based agents. However, they identify a gap in existing methods, which predominantly focus on local code information such as issues, classes, and functions. This local focus leads to a failure in capturing the global context and interdependencies within software systems, which are crucial for complex tasks in ASE.

2. RepoUnderstander Overview

The proposed RepoUnderstander method aims to address these limitations by developing a comprehensive understanding of whole repositories. The paper outlines several steps:

1. Repository Knowledge Graph Construction: A hierarchical tree structure is constructed from the repository, summarizing essential code snippets and their interdependencies.
2. Monte Carlo Tree Search (MCTS) Strategy: An exploration strategy based on MCTS is deployed to navigate the repository knowledge graph, focusing on nodes with high relevance scores.
3. Information Utilization and Patch Generation: Agents are guided to summarize and analyze the collected information, ultimately generating patches to resolve real-world GitHub issues.

Key Methodological Insights

The approach leverages several technical innovations:

• Top-down Repository Knowledge Graph Construction: By organizing repository information into a hierarchical structure, the method significantly reduces complexity, making it easier for agents to navigate and understand the code context.
• MCTS for Repository Exploration: The use of MCTS represents a nuanced strategy for effective repository understanding. By simulating multiple paths and evaluating reward scores, the method narrows down the search space to focus on the most relevant areas.
• In-context Learning and Chain-of-Thought for Reward Evaluation: These techniques enable a nuanced assessment of node relevance, ensuring that the agents can effectively prioritize important information.

Empirical Validation

The paper's empirical section demonstrates the method’s performance using the SWE-bench Lite benchmark, showing an 18.5% relative improvement over the current leading method, SWE-agent. Crucially, RepoUnderstander achieved a problem-solving rate of 21.33%, the highest among competitive baselines. These results underscore the effectiveness of understanding the global context within repositories for ASE tasks.

Practical and Theoretical Implications

Practical Implications

RepoUnderstander’s ability to understand and navigate large codebases can significantly enhance the efficiency and accuracy of ASE tasks such as fault localization and program repair. The method’s applicability to real-world GitHub issues highlights its practical relevance and potential for widespread adoption in the software engineering industry.

Theoretical Implications

The framework demonstrates a shift from local to global understanding in software repositories, suggesting that future ASE methods should prioritize holistic repository comprehension. This could lead to more sophisticated models capable of tackling increasingly complex software engineering challenges.

Speculative Outlook

As LLMs and ASE capabilities evolve, future developments may integrate RepoUnderstander with runtime feedback mechanisms. Combining comprehensive repository understanding with dynamic execution feedback could further enhance the robustness and accuracy of ASE tools, paving the way for fully autonomous software maintenance and development systems.

Conclusion

The paper presents a robust method for whole repository understanding, significantly contributing to the ASE field. RepoUnderstander’s innovative use of hierarchical knowledge graphs, MCTS, and advanced LLM techniques sets a new standard for future research, emphasizing the critical role of global context in complex software engineering tasks.