Edge-free but Structure-aware: Prototype-Guided Knowledge Distillation from GNNs to MLPs

Edge-free but Structure-aware: Prototype-Guided Knowledge Distillation from GNNs to MLPs

Introduction

Graph Neural Networks (GNNs) have shown stellar performance in handling non-Euclidean data, especially for tasks related to graph machine learning such as node classification. However, their high latency due to the neighborhood aggregation operation makes their use in real-world applications challenging. On the other hand, Multi-Layer Perceptrons (MLPs) offer low-latency solutions but fall short on graph tasks due to their inability to capture graph structural information. This paper introduces a novel method, Prototype-Guided Knowledge Distillation (PGKD), which enables the distillation of GNNs into MLPs while capturing graph structure in an edge-free manner.

PGKD Methodology

PGKD is underpinned by the identification and categorization of graph edges into intra-class and inter-class edges to understand their impact on GNNs. The method utilizes class prototypes—typical embedding vectors representing each class—to distill graph structural knowledge from GNNs to MLPs without requiring graph edge information. Specifically, PGKD includes:

• Intra-class loss: Encourages nodes of the same class to be closer to their class prototype, capturing homophily in an edge-free setting.
• Inter-class loss: Aligns the relative distances between different class prototypes as learned by GNNs, thus preserving class separation discovered by the GNN teachers in the distilled MLPs.

Experimental Results

The efficacy of PGKD is validated through experiments on various graph benchmarks, demonstrating not only its robustness and effectiveness but also its superiority over existing methods. PGKD shows marked improvements over GLNN, a baseline edge-free model, across both transductive and inductive settings on multiple datasets including Cora, Citeseer, and Pubmed. Ablation studies reinforce the importance of both intra-class and inter-class losses in achieving desirable performance.

Discussion and Analysis

Further analyses probe into PGKD's robustness against noisy node features, its performance across different inductive split ratios, and the impact of MLP configurations on model outcomes. PGKD consistently outperforms baselines across different levels of noise and configurations, highlighting its flexibility and robustness. Moreover, t-SNE visualizations of node representations offer insights into how PGKD effectively captures graph structure, enabling MLPs to achieve competitive accuracy to their GNN counterparts.

Implications and Future Directions

This paper's introduction of PGKD marks a significant contribution towards bridging the gap between the structural awareness of GNNs and the low-latency advantage of MLPs. The method's edge-free and structure-aware characteristics expand the potential application of MLPs in graph machine learning tasks. Future research could extend this methodology to a broader range of graph tasks beyond node classification and delve into optimizing the prototype generation for enhanced performance and interpretability.

Conclusion

Prototype-Guided Knowledge Distillation (PGKD) emerges as a novel and effective approach for distilling GNNs into MLPs, preserving graph structural information without the need for edge data. Its robustness, coupled with empirical improvements over existing methods, positions PGKD as a promising direction for future research in graph machine learning, particularly in applications where low latency is paramount.