Understanding Large Language Model Based Fuzz Driver Generation

Fuzz drivers are necessary for library API fuzzing. Automatic fuzz driver generation is challenging since it requires generating high quality API usage code which is correct and robust. Large language model based fuzz driver generator is a promising direction. Compared to traditional program analysis based generators, it is a text-based approach which is more lightweight and general. It can easily leverage various sources of API usage information for generation and generate human-friendly code. Nonetheless, there still lacks the basic understanding on this direction. To fill this gap, we did a study aiming the core issues of using LLMs on effective fuzz driver generation. For systematic understanding, 5 query strategies are designed and analyzed from basic to enhanced. For evaluation in scale, we built a semi-automatic framework, containing a quiz with 86 driver generation questions collected from 30 popular C projects, and a set of criteria for precise driver effectiveness validation. In total, 189,628 fuzz drivers using 0.22 billion tokens are generated and evaluated. Besides, generated drivers were compared with industrial used ones to obtain practical insights (3.12 CPU year fuzzing experiments). Our study revealed: 1) LLM-based generation has shown promising practicality. 64% questions can be solved entirely automatically and the number rises to 91% if manual semantic validators are incorporated. Moreover, the generated drivers exhibited competitive performance to those commonly employed in the industry; 2) LLMs struggle to generate fuzz drivers that require complex API usage specifics. Three key designs can help: repeatedly querying, querying with examples, and iteratively querying. Combining them yields a dominant strategy; 3) Significant rooms for improvement are still left, such as automatic semantic correctness validation, API usage expansion, and semantic oracle generation.