Fixing Hardware Security Bugs with Large Language Models (2302.01215v1)

Abstract: Novel AI-based code-writing LLMs such as OpenAI's Codex have demonstrated capabilities in many coding-adjacent domains. In this work we consider how LLMs maybe leveraged to automatically repair security relevant bugs present in hardware designs. We focus on bug repair in code written in the Hardware Description Language Verilog. For this study we build a corpus of domain-representative hardware security bugs. We then design and implement a framework to quantitatively evaluate the performance of any LLM tasked with fixing the specified bugs. The framework supports design space exploration of prompts (i.e., prompt engineering) and identifying the best parameters for the LLM. We show that an ensemble of LLMs can repair all ten of our benchmarks. This ensemble outperforms the state-of-the-art Cirfix hardware bug repair tool on its own suite of bugs. These results show that LLMs can repair hardware security bugs and the framework is an important step towards the ultimate goal of an automated end-to-end bug repair framework.

• The paper presents a novel framework that leverages LLMs and detailed prompt engineering to detect and repair hardware security bugs in Verilog HDL.
• It employs a multi-component methodology—combining static analysis, repair generation, and simulation-based evaluation—to validate functional and security compliance.
• Experimental results show that robust models like code-davinci at lower temperature settings yield consistent and accurate repairs, outperforming traditional tools such as CirFix.

Fixing Hardware Security Bugs with LLMs

Recent advancements in AI-based tools have enabled innovative applications in various domains, including automated bug fixing in software. The paper "Fixing Hardware Security Bugs with LLMs" explores utilizing LLMs for detecting and repairing security-related bugs in hardware designs, specifically focusing on Verilog HDL. The paper provides both theoretical insights and practical applications of LLMs in the field of hardware bug repair.

Approach and Framework

The core approach utilized in the paper involves constructing a specialized framework that assesses the performance of any LLM aimed at fixing specified hardware security bugs. The framework consists of four main components:

1. Sources: This involves gathering a set of domain-representative hardware security bugs from various systems, such as Verilog files demonstrating both bugs and their functional behavior.
2. Detector: A static analysis tool used primarily to detect bugs and classify them using CWEs. This tool is essential for automating the bug identification process.
3. Repair Generator: This is the essence of using LLMs; the flawed code is presented to the LLM with instructions, and repair suggestions are generated.
4. Evaluator: This component uses simulation tools (like ModelSim in this paper) to verify the correctness of LLM-generated repairs through functional and security evaluations.

   Figure 1: Overview of the framework used in our experiments It is broken down into 4 main components. Sources are the designs containing bugs. Detector localizes the bug (for bugs 8-10). Repair generator contains the LLM which generates the repairs. Evaluator verifies the success of the repair.

Implementation Details

Prompt Engineering

Prompt engineering is crucial when using LLMs for hardware bug repair. Variations in the instruction sets (prompts) significantly impact the success rate of bug repairs. The paper experimented with five instruction variations from minimal assistance ("no instructions") to detailed bug-fixing guidance expressed in pseudo-code.

Model and Temperature Variations

The experiments used multiple models, including OpenAI's code-davinci and code-cushman, as well as the open-source CodeGen model. Temperature settings were tested across a range from 0.1 to 0.9, impacting the determinism and creativity of the generated repairs.

Evaluation Metrics

Repairs were assessed based on two primary criteria:

• Functional Evaluation: Ensuring the repaired code passes all functional test cases.
• Security Evaluation: Verifying the repaired code adheres to specified security measures using both testbenches and static analysis tools.

  Figure 2: Results showing the performance of each LLM across all bugs in the form of heatmaps. Each small square shows the number of correct repairs for the corresponding instruction variation and temperature of the LLM. The maximum possible value is 200. A higher value indicates more success in generating repairs and is shaded in a darker color.

Experimental Results and Findings

The paper demonstrated that LLMs could effectively repair hardware security bugs, achieving varying degrees of success dependent on the choice of LLM, instruction variation, and temperature. Some key findings include:

• Prompt Detail: More detailed prompts with specific instructions led to higher success rates.
• Model Selection: The code-davinci models outperformed other models, showcasing the importance of using robust LLMs for complex tasks like hardware bug repair.
• Temperature Settings: Lower temperatures (e.g., 0.1) yielded more consistent results, likely due to reduced variance in responses.
• Comprehensive Repair Framework: The evaluated framework successfully localized bugs and produced repairs, outperforming existing tools such as CirFix on shared benchmarks.

  Figure 3: Number of correct repairs per bug. The number above each bar shows the sum of successful repairs across all LLMs for the corresponding bug. The maximum possible value is 2000. A higher value indicates that the bug was repaired more times.

Conclusion

The paper demonstrates that LLMs hold significant potential in automatically repairing hardware security bugs, complementing existing techniques. The framework and methodologies presented provide a basis for further exploration in hardware security and AI-based bug fixing. Future work is expected to improve upon the localization of bugs and refine LLM-based methodologies to ensure higher efficiency and broader practical applications in the semiconductor industry.