AI-assisted coding: Experiments with GPT-4 (2304.13187v1)

Abstract: AI tools based on LLMs have acheived human-level performance on some computer programming tasks. We report several experiments using GPT-4 to generate computer code. These experiments demonstrate that AI code generation using the current generation of tools, while powerful, requires substantial human validation to ensure accurate performance. We also demonstrate that GPT-4 refactoring of existing code can significantly improve that code along several established metrics for code quality, and we show that GPT-4 can generate tests with substantial coverage, but that many of the tests fail when applied to the associated code. These findings suggest that while AI coding tools are very powerful, they still require humans in the loop to ensure validity and accuracy of the results.

• The paper shows that GPT-4 effectively generates and improves Python code, though nearly 40% of initial attempts need further human-prompted corrections.
• The experiments reveal that GPT-4's refactored code consistently achieves better quality metrics, including reduced flake8 issues and enhanced maintainability.
• The study highlights that despite high test coverage from automated test creation, human oversight remains crucial due to frequent test failures.

Overview of AI-Assisted Coding with GPT-4

Researchers conducted a series of experiments to evaluate GPT-4's capabilities in generating and improving computer code. Despite the considerable capacity of GPT-4 to assist in coding tasks, it became apparent that human validation remains essential to ensure accurate performance. This evaluation not only sheds light on the proficiency of GPT-4 in coding but also highlights its current limitations, suggesting that AI coding assistants, although powerful, are not completely autonomous.

Experimentation with Data Science Problems

The first set of experiments focused on using GPT-4 to solve data science problems. GPT-4 was tasked with generating usable code based on various prompts. While it produced successful solutions in a substantial majority of attempts, nearly 40% were successful on their first prompt. However, a significant portion of these outcomes required additional prompts to address issues such as the use of outdated functions or incorrect API labeling. In several instances, the team was unable to resolve issues within a reasonable timeframe, revealing the need for human intervention in debugging and updating the AI's output.

Code Refactoring Analysis

When assessing the refactoring capabilities of GPT-4, researchers compared over 2000 examples of Python code from GitHub with GPT-4's refactoring outputs. Analysis revealed that refactored code had fewer issues according to the flake8 linter and overall better code quality metrics such as logical lines of code and maintainability index. Even though GPT-4 improved the code's readability and standards compliance, human oversight was still required for maximum effectiveness, suggesting a potential role for GPT-4 in enhancing code quality in conjunction with other programming tools.

Automatic Test Creation Performance

Researchers also tested GPT-4's ability to write tests for its generated code. Despite high test coverage, a majority of the automated tests failed upon execution. These failures often required extensive debugging to discern whether the fault lay with the code or the test itself, stressing the indispensable role of human expertise and oversight in the test verification process.

Implications and Conclusions

In conclusion, these experiments confirmed GPT-4's sophisticated ability to generate Python code, aligning with previous findings. Nevertheless, the observed prevalence of errors accentuates the vital role of human programmers in the development process. The paper informs us that while GPT-4 can aid researchers in producing functional and maintainable code, it cannot replace the need for human judgment and domain-specific knowledge. Thus, while AI coding assistants like GPT-4 are game-changing tools, they must be used in concert with human expertise to be truly effective.

The complete details and materials related to this paper, along with the specific prompts used, can be accessed through their public GitHub repository, ensuring reproducibility and transparency in scientific research.