Weisfeiler-Leman at the margin: When more expressivity matters (2402.07568v2)

Abstract: The Weisfeiler-Leman algorithm ($1$-WL) is a well-studied heuristic for the graph isomorphism problem. Recently, the algorithm has played a prominent role in understanding the expressive power of message-passing graph neural networks (MPNNs) and being effective as a graph kernel. Despite its success, $1$-WL faces challenges in distinguishing non-isomorphic graphs, leading to the development of more expressive MPNN and kernel architectures. However, the relationship between enhanced expressivity and improved generalization performance remains unclear. Here, we show that an architecture's expressivity offers limited insights into its generalization performance when viewed through graph isomorphism. Moreover, we focus on augmenting $1$-WL and MPNNs with subgraph information and employ classical margin theory to investigate the conditions under which an architecture's increased expressivity aligns with improved generalization performance. In addition, we show that gradient flow pushes the MPNN's weights toward the maximum margin solution. Further, we introduce variations of expressive $1$-WL-based kernel and MPNN architectures with provable generalization properties. Our empirical study confirms the validity of our theoretical findings.

• The paper introduces theoretical bounds linking WL-based kernel VC dimensions with margin theory to quantify expressivity in graph models.
• It demonstrates that integrating subgraph information increases the effective margin, which enhances generalization on both synthetic and real-world datasets.
• Empirical studies confirm that MPNNs augmented with subgraph encoding outperform traditional models, bridging theoretical insights with practical performance.

Weisfeiler--Leman at the Margin: When More Expressivity Matters

Abstract

This paper investigates the Weisfeiler--Leman algorithm's role in characterizing the expressive power of message-passing graph neural networks (MPNNs) and graph kernels, particularly focusing on the relationships between expressivity and generalization performance. The Weisfeiler--Leman algorithm, a known heuristic for the graph isomorphism problem, has seen applications in both MPNN architectures and as the foundation for graph kernels. While extensions have been developed to address its limitations, the connection between increased expressivity and generalization remains ambiguous. This work extends the Weisfeiler--Leman algorithm and corresponding MPNNs by integrating subgraph information and employing classical margin theory to examine conditions under which increased expressivity aligns with improved generalization performance. The paper further establishes theoretical bounds, demonstrating how gradient flow pushes MPNN weights toward a maximum margin solution and presents empirical results supporting these theories.

Introduction

Graph-structured data is ubiquitous across several disciplines, from bioinformatics and cheminformatics to social network analysis. Machine learning methods for graphs, such as graph kernels and MPNNs, leverage the structure of graph data to achieve state-of-the-art results. In particular, the Weisfeiler--Leman (WL) algorithm has been instrumental in defining the expressiveness of these methods. However, due to its limitations in distinguishing non-isomorphic graphs, several more expressive variants of WL and corresponding enhancements to MPNNs have been proposed. Despite their empirical success, a clear theoretical rationale linking expressivity to performance improvements is still largely unexplored.

Main Contributions

VC Dimension and Margin Theory

The theoretical contribution of this paper is primarily grounded in the application of classical margin theory to graph isomorphism problems. Specifically, the paper derives upper and lower bounds for the Vapnik--Chervonenkis (VC) dimension of Weisfeiler--Leman-based kernels and MPNNs parameterized by the margin separating the data. Several key results were established:

• Weisfeiler--Leman (WL) Kernel: The paper demonstrates that the VC dimension of the WL kernel can be tightly bounded. For a fixed number of iterations $T$, the VC dimension is dependent on the number of vertices $n$ and the margin $\lambda$.
• More Expressive Variants: By extending the WL algorithm with subgraph information (termed WL$_F$) and employing the Weisfeiler--Leman optimal assignment kernel (WLOA), the authors show that the expressivity of these variants can significantly increase the effective margin, leading to better generalization properties in certain cases.
• MPNN Architectures: Analogous to the findings for kernel methods, MPNN architectures augmented with subgraph information ($\text{MPNN-F}$) also exhibit improved generalization properties when viewed through the lens of margin theory.

Empirical Studies

The empirical validation provided in the paper involves several graph classification benchmark datasets. The experimental results confirm the theoretical predictions about the relationships between expressivity, margin, and generalization:

• Synthetic Datasets Verification: The authors demonstrated data distributions where more expressive WL-based methods could linearly separate data that the classic WL kernel could not.
• Generalization Improvement: Across several real-world datasets, the augmented MPNN architectures (MPNN-F) generally outperformed the baseline MPNNs. The introduction of subgraph encoding led to an increased margin which theoretically correlates with improved generalization performance.

Practical and Theoretical Implications

The theoretical bounds and empirical results presented in this work bridge an essential gap in understanding the benefits of enhanced expressivity in MPNNs and graph kernels. The reliance on margin theory provides a robust framework to evaluate and predict the generalization performance of graph-based machine learning models. Practically, these insights can guide the design of more effective graph neural networks and kernel methods by ensuring that expressive power is utilized in a manner that leads to better generalization.

Future Directions

Future research avenues based on this work could involve the exploration of other types of subgraph features beyond those utilized here to further enhance the expressivity of MPNNs and kernels. Moreover, investigating the convergence properties of gradient descent methods for these enhanced architectures and their practical implementation in a broader array of real-world applications would be valuable.

Conclusion

This paper makes significant strides in linking the expressivity of MPNNs and graph kernels to their generalization performance through margin theory. By demonstrating that more expressive variants of the Weisfeiler--Leman algorithm can enhance performance, the work paves the way for further theoretical and practical advancements in graph-based machine learning. The paper's theoretical and empirical analyses provide a comprehensive understanding of when and how increased expressivity translates to improved generalization, offering a more nuanced perspective for the development of future graph learning methods.