LLM4VV: Developing LLM-Driven Testsuite for Compiler Validation (2310.04963v3)

Abstract: LLMs are a new and powerful tool for a wide span of applications involving natural language and demonstrate impressive code generation abilities. The goal of this work is to automatically generate tests and use these tests to validate and verify compiler implementations of a directive-based parallel programming paradigm, OpenACC. To do so, in this paper, we explore the capabilities of state-of-the-art LLMs, including open-source LLMs -- Meta Codellama, Phind fine-tuned version of Codellama, Deepseek Deepseek Coder and closed-source LLMs -- OpenAI GPT-3.5-Turbo and GPT-4-Turbo. We further fine-tuned the open-source LLMs and GPT-3.5-Turbo using our own testsuite dataset along with using the OpenACC specification. We also explored these LLMs using various prompt engineering techniques that include code template, template with retrieval-augmented generation (RAG), one-shot example, one-shot with RAG, expressive prompt with code template and RAG. This paper highlights our findings from over 5000 tests generated via all the above mentioned methods. Our contributions include: (a) exploring the capabilities of the latest and relevant LLMs for code generation, (b) investigating fine-tuning and prompt methods, and (c) analyzing the outcome of LLMs generated tests including manually analysis of representative set of tests. We found the LLM Deepseek-Coder-33b-Instruct produced the most passing tests followed by GPT-4-Turbo.

• The paper presents LLM4VV, an approach that leverages LLMs to generate validation tests for OpenACC compiler implementations.
• It employs a mix of fine-tuning, prompt engineering, and retrieval-augmented generation to optimize test creation and address specification challenges.
• Performance evaluations reveal that Deepseek-Coder-33b-Instruct achieved the highest pass rates, highlighting both effective automation and areas for improvement.

Overview of "LLM4VV: Developing LLM-Driven Testsuite for Compiler Validation"

The paper, "LLM4VV: Developing LLM-Driven Testsuite for Compiler Validation" (2310.04963), presents a novel approach using LLMs for generating validation and verification (V&V) testsuites tailored for OpenACC compiler implementations. The authors leverage both open-source models like Meta's Codellama and Phind's fine-tuned Codellama, as well as proprietary models such as OpenAI's GPT-3.5-Turbo and GPT-4-Turbo, to automate test generation. The paper explores various methodologies including fine-tuning of LLMs, prompt engineering, and retrieval-augmented generation (RAG) to optimize test creation. Through extensive evaluation across more than 5000 generated tests, it was found that the Deepseek-Coder-33b-Instruct model outperformed others in producing passing tests, followed closely by GPT-4-Turbo.

Motivation and Methodology

Motivation

The increasing complexity in compiler implementations for directive-based programming paradigms, such as OpenACC, necessitates rigorous validation to ensure compliance with specifications. Manual tests are labor-intensive and prone to human error, especially given the frequent updates to programming model specifications. Thus, automating this process using LLMs promises significant efficiency gains, reduced overhead in test generation, and the potential for broad applicability across different versions of programming models like OpenACC and the evolving OpenMP.

Implementation Details

Selection of LLMs and Benchmarks

The paper evaluates both open-source and closed-source LLMs. The choice of models is based on benchmarks like HumanEval and MBPP+, which test code generation capabilities. The focus was on models pre-trained on large code datasets and then potentially fine-tuned with OpenACC-specific data using parameter-efficient techniques.

Prompt Engineering and Fine-tuning

The paper investigates various prompt engineering techniques such as template-based prompts, RAG, one-shot and expressive prompts. Fine-tuning is executed using a curated dataset, comprising OpenACC-specific instructions to tailor the LLM responses. Fine-tuning allows LLMs to generate tests effectively without needing detailed examples in prompts.

Retrieval-Augmented Generation (RAG)

RAG is implemented to provide LLMs with the latest OpenACC specification details during test case generation, addressing issues of model hallucination and outdated information that may arise if LLMs are not trained on the latest datasets.

Results and Analysis

Stage-wise Evaluation

The research involved a staged approach to evaluate LLM performance. Initial stages explored the effectiveness of different prompting strategies, while later stages included manual analysis of test outputs:

• Stage 1 focused on identifying the most effective LLM and prompt method combinations.
• Stage 2 tested a more extensive set of cases, including variations in base languages (C, C++, Fortran) and combinations of constructs and clauses.
• Stage 3 included manual analysis to identify the causes of errors and assess the accuracy of "passing" test cases.

Key Findings

• Performance Metrics: Deepseek-Coder-33b-Instruct achieved the highest pass rates, with substantial room for improvement in the correctness metric.
• Error Analysis: A significant number of errors arose from incorrect OpenACC implementations within tests or hallucinated routines, indicating potential areas for improvement in LLM training datasets and prompt design.
• Prompt Efficiency: Expressive prompts with RAG and templates showed significant promise in improving the quality of generated tests compared to simpler one-shot prompts.
• Language Disparities: There was variance in performance across different programming languages, with Fortran showing the lowest pass rates, suggesting a need for more language-specific training data.

Implications and Future Work

The research underscores the feasibility of employing LLMs for automated test generation in high-performance computing contexts, albeit with current limitations in perfect accuracy and understanding of specifications. Moving forward, integrating feedback loops for LLMs to iteratively improve output based on error analysis is crucial. Additionally, extending these methodologies to other programming models like OpenMP and emerging languages could broaden the utility of LLM-based testsuites.

Improving RAG techniques to retrieve more contextual information and refining the balance between prompt complexity and specificity will play a critical role in further advancing LLM capabilities in this domain. The fusion of these AI-driven approaches with traditional software verification processes holds promise for significantly enhancing the reliability and maintainability of compiler implementations.