AutoCoder: Enhancing Code Large Language Model with \textsc{AIEV-Instruct}

AutoCoder: Enhancing Code Large Language Model with AIEV-Instruct

The paper titled "AutoCoder: Enhancing Code Large Language Model with AIEV-Instruct" introduces AutoCoder, a novel LLM for code generation, trained using an innovative dataset annotation method named AIEV-Instruct. This paper aims to address two prominent challenges in the existing methodologies for code generation with LLMs: the reliance on expensive proprietary models for annotation and the transmission of incorrect knowledge during the distillation process from teacher models to student models.

Motivation

Code generation is increasingly becoming a critical tool in modern software development, as it can significantly increase productivity, reduce errors, and support the development of complex systems. Various closed-source and open-source models have shown promise in this domain, but there remain key challenges in achieving high accuracy without excessive costs and dependencies on closed-source models.

Methodology: AIEV-Instruct

AIEV-Instruct, which stands for Agent-Interaction and Execution-Verified Instruction, is the core methodological innovation presented in this paper. It revolutionizes the process of creating high-quality code instruction datasets through simulated dialogues between two agents: a questioner and a programmer.

The AIEV-Instruct process is separated into two primary stages:

1. Teaching Stage: During this stage, a proprietary model is used as the "teacher" to generate code snippets and corresponding unit tests from open-source fragments. The dialogue between the agents ensures diverse and multi-turn interaction, capturing a comprehensive set of programming scenarios. The proprietary model generates an initial code and unit test, which are then verified by execution. Errors are fed back into the system to iteratively improve the generated code until it passes all unit tests.
2. Self-Learning Stage: Once AutoCoder surpasses proprietary models in accuracy on the test sets, it transitions to the Self-Learning Stage, where AutoCoder itself assumes both the questioner and programmer roles. This self-sustaining loop significantly reduces the dependence on expensive proprietary models and enables autonomous learning and dataset generation.

AutoCoder Training and Evaluation

Using the AIEV-Instruct dataset, AutoCoder was trained in two configurations: AutoCoder (33B parameters) and AutoCoder-S (6.7B parameters). Training was conducted using the Deepseek-Coder as the base model. AutoCoder introduces a new feature in its code interpreter environment, which allows it to execute bash commands for external package installations, addressing a significant limitation in existing models like GPT-4 and GPT-4 Turbo.

Experimental Results

The experimental results presented in the paper are notable for several reasons:

1. HumanEval Performance: AutoCoder achieved a Pass@1 score of 90.9%, surpassing the performance of GPT-4 Turbo (90.2%) and all current state-of-the-art models.
2. Benchmarks Across Diverse Datasets: AutoCoder was evaluated on multiple datasets including HumanEval+, MBPP, MBPP+, MultiPL-E, and DS-1000. Consistently, it performed admirably, ranking among the top across these benchmarks:

• HumanEval+: 78% Pass@1, second only to GPT-4 Turbo and CodeQwen1.5-Chat
• MBPP and MBPP+: 82.5% and 70.6% Pass@1 respectively
• Multilingual Performance: AutoCoder showed robust capabilities across several programming languages, achieving 68.9% Pass@1 in both C++ and JavaScript.
• Data Science Code Generation (DS-1000): AutoCoder's performance was second only to GPT-4 Turbo with an overall Pass@1 exceeding 45%.

Implications and Future Directions

The implications of this research extend both practically and theoretically. Practically, AutoCoder offers a more versatile and autonomous approach to high-quality code generation, reducing costs and dependence on closed-source models. Theoretically, the AIEV-Instruct method presents a robust framework for iterative learning and validation in dataset generation, potentially applicable to other domains beyond code generation.

The paper's results suggest several avenues for future work:

• Extension to Other Programming Languages: Future work could explore the application of AIEV-Instruct in generating datasets for languages beyond those tested.
• Integration with IDEs: Further integration of AutoCoder within Integrated Development Environments (IDEs) to provide real-time assistance and corrections to developers.
• Expansion of the Dialogue-Based Annotation: Expanding the annotation methodology to more complex, multi-step programming tasks.

In summary, this paper makes significant contributions to the field of code generation with LLMs by presenting AutoCoder and the AIEV-Instruct methodology. It demonstrates improved performance and versatility over existing models, suggesting a path towards more autonomous and cost-effective code generation solutions.