CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Models (2302.04012v2)

Abstract: LLMs for automatic code generation have achieved breakthroughs in several programming tasks. Their advances in competition-level programming problems have made them an essential pillar of AI-assisted pair programming, and tools such as GitHub Copilot have emerged as part of the daily programming workflow used by millions of developers. The training data for these models is usually collected from the Internet (e.g., from open-source repositories) and is likely to contain faults and security vulnerabilities. This unsanitized training data can cause the LLMs to learn these vulnerabilities and propagate them during the code generation procedure. While these models have been extensively assessed for their ability to produce functionally correct programs, there remains a lack of comprehensive investigations and benchmarks addressing the security aspects of these models. In this work, we propose a method to systematically study the security issues of code LLMs to assess their susceptibility to generating vulnerable code. To this end, we introduce the first approach to automatically find generated code that contains vulnerabilities in black-box code generation models. To achieve this, we present an approach to approximate inversion of the black-box code generation models based on few-shot prompting. We evaluate the effectiveness of our approach by examining code LLMs in generating high-risk security weaknesses. Furthermore, we establish a collection of diverse non-secure prompts for various vulnerability scenarios using our method. This dataset forms a benchmark for evaluating and comparing the security weaknesses in code LLMs.

• The paper presents an automated approach using few-shot prompting to invert code models and expose security vulnerabilities.
• It employs FS-Code, FS-Prompt, and OS-Prompt strategies to simulate insecure code generation and evaluate diverse sampling outputs.
• The created CodeLMSec benchmark systematically assesses and ranks code language models based on their propensity to generate vulnerable code.

CodeLMSec Benchmark: Evaluating Vulnerabilities in Code LLMs

The paper "CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code LLMs" addresses the critical issue of security vulnerabilities in code generated by LLMs. With the increasing reliance on tools like GitHub Copilot for AI-assisted programming, understanding the security implications of automatically generated code becomes essential.

Introduction to CodeLMSec

The research highlights the challenges posed by unsanitized data used in training LLMs, which often incorporates security vulnerabilities present in open-source repositories. The lack of comprehensive studies on the security aspect of LLM-generated code underscores the need for systematic approaches to evaluate these models beyond functional correctness.

Methodology: Model Inversion and Few-Shot Prompting

The core contribution of this paper is an automated approach that approximates the inverse of black-box models using few-shot prompting to discover security vulnerabilities. This method involves:

• Model Inversion: The process is visualized as approximating the inverse of code generation models to predict prompts that result in vulnerabilities.
• Few-Shot Learning: By leveraging a few examples of known vulnerable code (Figure 1), the model is guided to generate similar scenarios that exploit the same vulnerabilities.

  Figure 1: We systematically find vulnerabilities and associated prompts by approximating the inverse of black-box code generation model $\mathbf{F}$.

  

  Figure 2: Overview of our proposed approach to automatically finding security vulnerability issues of the code generation models.

Implementation Strategy

The implementation involves three strategies: FS-Code, FS-Prompt, and OS-Prompt. FS-Code combines code examples with vulnerabilities, leading to these primary outcomes:

1. Non-Secure Prompt Generation: Using few-shot examples that include vulnerabilities, the model generates prompts conducive to creating insecure code.
2. Sampling Techniques: Strategies like nucleus sampling are employed to diversify output and explore the model's potential for generating vulnerable code across various sampling temperatures (Figure 3).

   Figure 3: Number of discovered vulnerable Python codes using different sampling temperatures.

Evaluation and Results

The paper presents extensive evaluations comparing vulnerabilities discovered across different CWEs in Python and C code. Notably, FS-Code and FS-Prompt exhibit substantial efficacy in generating code that mirrors the weaknesses present in the few-shot examples (Figure 4).

Figure 4: Number of discovered vulnerable Python codes using a different number of few-shot examples.

Benchmarking and Future Implications

A key outcome of this research is the creation of the CodeLMS benchmark dataset, designed to systematically assess and rank code models based on their propensity to output vulnerable code. This benchmark allows for the evaluation of current and future model versions, promoting continuous improvements in secure code generation practices.

Discussion

While demonstrating the scalability and effectiveness of their approach, the authors acknowledge limitations such as reliance on static analysis tools like CodeQL, which may not capture all potential vulnerabilities. Furthermore, this work lays the groundwork for future explorations into enhancing model reliability, potentially influencing model training methods to mitigate security risks.

Conclusion

This research provides a vital foundation for systematically evaluating security vulnerabilities in LLMs for code generation. By introducing methodologies for discovering vulnerabilities and establishing benchmarks, it sets a precedent for future endeavors to refine model outputs and enhance security in automatically generated code. The release of the CodeLMSec dataset provides a practical tool for developers and researchers to diagnose, compare, and improve the security integrity of code LLMs.