Harvesting Production GraphQL Queries to Detect Schema Faults (2112.08267v2)

Abstract: GraphQL is a new paradigm to design web APIs. Despite its growing popularity, there are few techniques to verify the implementation of a GraphQL API. We present a new testing approach based on GraphQL queries that are logged while users interact with an application in production. Our core motivation is that production queries capture real usages of the application, and are known to trigger behavior that may not be tested by developers. For each logged query, a test is generated to assert the validity of the GraphQL response with respect to the schema. We implement our approach in a tool called AutoGraphQL, and evaluate it on two real-world case studies that are diverse in their domain and technology stack: an open-source e-commerce application implemented in Python called Saleor, and an industrial case study which is a PHP-based finance website called Frontapp. AutoGraphQL successfully generates test cases for the two applications. The generated tests cover 26.9% of the Saleor schema, including parts of the API not exercised by the original test suite, as well as 48.7% of the Frontapp schema, detecting 8 schema faults, thanks to production queries.

• The paper presents AutoGraphQL, a tool that generates GraphQL API tests by monitoring and leveraging real-world queries from production systems to find schema faults.
• Evaluated on Saleor and Frontapp, AutoGraphQL successfully generated over 24,000 tests and detected 8 confirmed schema faults in Frontapp, demonstrating high fault detection efficacy.
• The approach provides a practical method to bridge the gap between API definition and actual usage, significantly improving the reliability of GraphQL APIs and aiding regression testing.

Overview of "Harvesting Production GraphQL Queries to Detect Schema Faults"

The paper "Harvesting Production GraphQL Queries to Detect Schema Faults" presents AutoGraphQL, a tool designed to automatically generate test cases for GraphQL APIs. It addresses the challenge of detecting schema faults, which occur when GraphQL queries return data incompatible with schema definitions. The proposed solution capitalizes on real-world usage of GraphQL APIs by intercepting queries in production systems to inform test generation.

Key Contributions and Methodology

AutoGraphQL introduces a novel approach by leveraging the naturally occurring GraphQL queries in live applications as a basis for generating automated tests. This method ensures that the generated tests cover real-world use cases, potentially exercising parts of the system overlooked by developer-written tests.

The methodology is built around two core phases:

1. Monitoring Production Queries: AutoGraphQL integrates with production systems to log GraphQL queries. This logging process captures both the query syntax and its execution context (specifically, the arguments used). This systematic interception provides a unique view of how the API is utilized in practice.
2. Automated Test Generation: Once a comprehensive set of queries is logged, AutoGraphQL generates test cases. Each test case is constructed by combining the intercepted query with a set of oracles derived from the schema. This allows verification of the response format against the schema, checking for data conformity in terms of required fields, correct types, and non-null constraints.

Experimental Evaluation

The tool was evaluated on two distinct case studies: an open-source e-commerce platform called Saleor, and a proprietary finance application named Frontapp. Through these evaluations, AutoGraphQL generated over 24,000 tests for Frontapp and 334 tests for Saleor.

Saleor

• Schema Coverage: AutoGraphQL achieved a schema coverage of 26.9% with 334 generated tests, introducing unique coverage areas not addressed by Saleor's developer-written test suite.
• Complementary Testing: The generated tests covered several query entry points not reached by the original tests, demonstrating the tool's capacity to enhance overall test suite effectiveness.

Frontapp

• Effective Fault Detection: Of the 24,049 tests generated, 157 failed, which led to the identification of 8 distinct schema faults in the API implementation. These were confirmed and resolved by the industrial partner.
• High Schema Coverage: Achieving 48.7% schema coverage, AutoGraphQL effectively highlighted areas of the schema engaged by real-world interactions, showcasing its potential in uncovering latent schema faults.

Practical and Theoretical Implications

The application of AutoGraphQL reveals significant potential for improving GraphQL API reliability in real-world settings. It provides a mechanism to bridge the gap between theoretical API behavior as defined by developers and practical usage as exercised by end-users. The main implication is a more robust detection of schema faults, facilitating regression testing and schema evolution.

Future Directions

The paper acknowledges challenges such as tailoring query interception across various server architectures, the execution overhead of large test suites, and adapting to schema evolution. Future work may focus on test suite minimization and prioritization, integration of mutation-based testing for state-altering queries, and strategies for maintaining test relevance amidst evolving schemas.

In conclusion, AutoGraphQL presents a significant step forward in leveraging production data to enhance automated testing for GraphQL APIs, providing a strong basis for ongoing research and development in API testing strategies.