On the "Naturalness" of Buggy Code (1506.01159v2)

Abstract: Real software, the kind working programmers produce by the kLOC to solve real-world problems, tends to be "natural", like speech or natural language; it tends to be highly repetitive and predictable. Researchers have captured this naturalness of software through statistical models and used them to good effect in suggestion engines, porting tools, coding standards checkers, and idiom miners. This suggests that code that appears improbable, or surprising, to a good statistical LLM is "unnatural" in some sense, and thus possibly suspicious. In this paper, we investigate this hypothesis. We consider a large corpus of bug fix commits (ca.~8,296), from 10 different Java projects, and we focus on its language statistics, evaluating the naturalness of buggy code and the corresponding fixes. We find that code with bugs tends to be more entropic (i.e., unnatural), becoming less so as bugs are fixed. Focusing on highly entropic lines is similar in cost-effectiveness to some well-known static bug finders (PMD, FindBugs) and ordering warnings from these bug finders using an entropy measure improves the cost-effectiveness of inspecting code implicated in warnings. This suggests that entropy may be a valid language-independent and simple way to complement the effectiveness of PMD or FindBugs, and that search-based bug-fixing methods may benefit from using entropy both for fault-localization and searching for fixes.

• The paper demonstrates that buggy code exhibits higher entropy than fixed code, linking unpredictability to defects.
• It employs statistical language models on 8,296 bug-fix commits from 10 Java projects to effectively localize bugs.
• Results reveal that entropy-based detection complements tools like PMD and FindBugs, optimizing traditional bug-finding methods.

Evaluating the "Naturalness" of Buggy Code

This paper investigates an intriguing hypothesis within the domain of software engineering: that buggy code tends to be "unnatural" in comparison to non-buggy code. The concept of "naturalness" draws from the field of NLP, where repetitive and predictable code sequences can be effectively modeled using statistical LLMs. Building on this premise, the authors explore whether improbable code sequences—those with higher entropy—indicate the presence of bugs. By leveraging the predictive power of LLMs, they evaluate if these models can successfully identify buggy lines of code across multiple large Java projects.

The research employs a robust methodology, analyzing approximately 8,296 bug-fix commits across 10 different Java projects to determine the naturalness of buggy versus fixed code lines. The paper demonstrates that buggy code exhibits higher entropy, indicative of a lack of predictability and regularity. The results reveal a significant reduction in entropy once the bugs are fixed, thereby suggesting that bug fixes tend to conform more closely to expected coding patterns.

The investigation proceeds by comparing the effectiveness of “naturalness” based bug localization to static bug-finding tools such as PMD and FindBugs. Through evaluation metrics like Area Under the Cost-Effectiveness Curve (AUCEC), the paper establishes that focusing on "unnatural" lines—those highlighted by LLMs as having high entropy—can be as effective as existing static analysis tools for inspecting potentially faulty code. Interestingly, the use of entropy not only complements the effectiveness of PMD or FindBugs but also enhances it, especially when used to reorder warnings.

The paper's implications are significant for both theoretical exploration and practical application. Theoretically, it underscores the potential of applying LLMs beyond traditional NLP tasks into software engineering. Practically, it introduces a supplementary tool for bug localization, which could optimize the performance of static bug finders and inform advanced bug-fixing methods.

Future work could see these findings being extended across different programming languages at scale, thus confirming or refuting the generalizability of the results. Further integration into automated debugging and repair systems also holds promise, with entropy potentially guiding search-based approaches to efficiently identify and propose higher probability bug fixes.

In conclusion, the research provides a compelling argument for the utilization of statistical LLMing in software engineering, highlighting entropy as a valuable metric for identifying unnatural—and potentially buggy—code lines. This paper contributes a novel perspective to bug localization, with the potential to inform future developments in AI-driven code analysis.