REST API Fuzzing by Coverage Level Guided Blackbox Testing (2112.15485v1)

Abstract: With the growth of web applications, REST APIs have become the primary communication method between services. In order to ensure system reliability and security, software quality can be assured by effective testing methods. Black box fuzz testing is one of the effective methods to perform tests on a large scale. However, conventional black box fuzz testing generates random data without judging the quality of the input. We implement a black box fuzz testing method for REST APIs. It resolves the issues of blind mutations without knowing the effectiveness by Test Coverage Level feedback. We also enhance the mutation strategies by reducing the testing complexity for REST APIs, generating more appropriate test cases to cover possible paths. We evaluate our method by testing two large open-source projects and 89 bugs are reported and confirmed. In addition, we find 351 bugs from 64 remote API services in APIs.guru. The work is in https://github.com/iasthc/hsuan-fuzz.

• The paper introduces a novel black box fuzzing method that leverages test coverage feedback to guide efficient REST API mutations.
• It implements HsuanFuzz, which uses grammar generation, pairwise testing, and strategic mutators to manage request sequencing and path dependencies.
• Evaluation reveals that HsuanFuzz achieves higher code coverage and superior bug detection compared to traditional REST API testing tools.

"REST API Fuzzing by Coverage Level Guided Blackbox Testing" (2112.15485)

Introduction and Objectives

The paper presents a methodology for implementing black box fuzz testing against REST APIs that enhances conventional methods by incorporating "Test Coverage Level" (TCL) as a feedback mechanism to guide and improve the quality of mutations. The primary objective is to address the shortcomings of classic black box fuzzing — which often suffers from inefficient blind mutations — by providing a systematic approach that leverages test coverage feedback to guide the fuzz testing process.

Core Contributions and Methodology

The key contributions of the research include:

1. Novel Strategy for Black Box Fuzzing: Introduces a black box fuzzing strategy that incorporates estimated code coverage as feedback, essentially creating an efficient mechanism to understand mutation impacts without internal code access.
2. Implementation of HsuanFuzz: A tool that accepts OpenAPI specifications and path dependencies to perform automated testing on REST APIs, making the process adaptable and scalable across diverse service architectures.
3. Resolution of Common REST API Testing Issues: The approach handles common testing complexities such as request sequencing, path dependencies, parameter validation, and access token management.
4. Pairwise Testing Technique: Adopted to limit the combinations of test parameters and accelerate the testing process, ensuring a more efficient coverage and bug detection routine.

Implementation Details

The fuzzing tool, "HsuanFuzz", is engineered to conduct black box testing by leveraging the following components:

• Grammar Generation: Utilizes OpenAPI specifications to generate initial grammar, maintaining the sequence of requests and addressing any path dependencies manually.
• Mutator Component: Employs both traditional string mutation techniques and strategically selects requests pairs for mutation to ensure optimal path exploration.
• Analysis Phase: Evaluates responses against TCL criteria to determine if a mutation has resulted in an enhanced coverage, which, if positive, persists the mutated input for further iterations.
• Error Reporting: The implementation details the error tracking and reporting capabilities that record occurrences where the server response indicates potential vulnerabilities (e.g., status code 500).

Evaluation and Results

The evaluation involved comparing the HsuanFuzz's performance against other known tools like RESTler, considering key metrics such as code coverage and error finding capabilities:

• Code Coverage: HsuanFuzz demonstrated superior abilities in achieving higher code coverage across tested services such as open-source e-commerce platforms and publicly available Web APIs.
• Bug Detection: Testing against both self-hosted and remote APIs indicated that the TCL-guided approach was more effective in discovering unique errors, confirming the robustness of the method.

Discussion and Implications

The paper posits that integrating coverage feedback in black box testing avails significant advantages, primarily in enhancing test effectiveness through guided input mutation. By focusing on high-level specifications, the strategy circumvents language barriers inherent in grey box testing methods and permits scalability across diverse networked services.

Conclusion

The research substantiates that a coverage level-guided methodology can substantially refine black box fuzz testing for REST APIs. Going forward, extending such methodologies to support protocols akin to CoAP for IoT devices could present considerable benefits, broadening the scope of automated testing paradigms in resource-constrained environments. Future research may explore intensive automatic identification methods or further enhance the granularity of test coverage levels to mirror code coverage accuracy more closely.