Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
124 tokens/sec
GPT-4o
8 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

SimMLP: Training MLPs on Graphs without Supervision (2402.08918v3)

Published 14 Feb 2024 in cs.LG, cs.AI, and cs.SI

Abstract: Graph Neural Networks (GNNs) have demonstrated their effectiveness in various graph learning tasks, yet their reliance on neighborhood aggregation during inference poses challenges for deployment in latency-sensitive applications, such as real-time financial fraud detection. To address this limitation, recent studies have proposed distilling knowledge from teacher GNNs into student Multi-Layer Perceptrons (MLPs) trained on node content, aiming to accelerate inference. However, these approaches often inadequately explore structural information when inferring unseen nodes. To this end, we introduce SimMLP, a Self-supervised framework for learning MLPs on graphs, designed to fully integrate rich structural information into MLPs. Notably, SimMLP is the first MLP-learning method that can achieve equivalence to GNNs in the optimal case. The key idea is to employ self-supervised learning to align the representations encoded by graph context-aware GNNs and neighborhood dependency-free MLPs, thereby fully integrating the structural information into MLPs. We provide a comprehensive theoretical analysis, demonstrating the equivalence between SimMLP and GNNs based on mutual information and inductive bias, highlighting SimMLP's advanced structural learning capabilities. Additionally, we conduct extensive experiments on 20 benchmark datasets, covering node classification, link prediction, and graph classification, to showcase SimMLP's superiority over state-of-the-art baselines, particularly in scenarios involving unseen nodes (e.g., inductive and cold-start node classification) where structural insights are crucial. Our codes are available at: https://github.com/Zehong-Wang/SimMLP.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (83)
  1. Deep variational information bottleneck. In ICLR, 2017.
  2. Learning representations by maximizing mutual information across views. In NeurIPS, 2020.
  3. Noise2self: Blind denoising by self-supervision. In ICML, 2019.
  4. Mutual information neural estimation. In ICML, 2018.
  5. A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses. In ECCV, 2020.
  6. The ssl interplay: Augmentations, inductive bias, and generalization. In ICML, 2023.
  7. Deep clustering for unsupervised learning of visual features. In ECCV, 2018.
  8. Unsupervised learning of visual features by contrasting cluster assignments. NeurIPS, 2020.
  9. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In AAAI, 2020a.
  10. On graph neural networks versus graph-augmented MLPs. In ICLR, 2021.
  11. A simple framework for contrastive learning of visual representations. In ICML, 2020b.
  12. Big self-supervised models are strong semi-supervised learners. NeurIPS, 2020c.
  13. VQ-GNN: A universal framework to scale up graph neural networks using vector quantization. In NeurIPS, 2021.
  14. Node representation learning in graph via node-to-neighbourhood mutual information maximization. In CVPR, 2022.
  15. Graph neural networks with learnable structural and positional representations. In ICLR, 2022.
  16. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2019.
  17. Combining neural networks with personalized pagerank for classification on graphs. In ICLR, 2019.
  18. Neural message passing for quantum chemistry. In ICML, 2017.
  19. Bootstrap your own latent-a new approach to self-supervised learning. In NeurIPS, 2020.
  20. Linkless link prediction via relational distillation. In ICML, 2023.
  21. Deep learning with limited numerical precision. In ICML, 2015.
  22. Inductive representation learning on large graphs. In NeurIPS, 2017.
  23. Learning both weights and connections for efficient neural network. In NeurIPS, 2015.
  24. MLPInit: Embarrassingly simple GNN training acceleration with MLP initialization. In ICLR, 2023.
  25. Pre-training graph neural networks for cold-start users and items representation. In WSDM, 2021.
  26. Contrastive multi-view representation learning on graphs. In ICML, 2020.
  27. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
  28. Using self-supervised learning can improve model robustness and uncertainty. In NeurIPS, 2019.
  29. Distilling the knowledge in a neural network. arXiv, 2015.
  30. Measuring and improving the use of graph information in graph neural networks. In ICLR, 2019.
  31. Graphmae: Self-supervised masked graph autoencoders. In KDD, 2022.
  32. Open graph benchmark: Datasets for machine learning on graphs. In NeurIPS, 2020.
  33. Graph-mlp: Node classification without message passing in graph. arXiv, 2021.
  34. Towards the generalization of contrastive self-supervised learning. In ICLR, 2023.
  35. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In CVPR, 2018.
  36. Understanding dimensional collapse in contrastive self-supervised learning. In ICLR, 2022.
  37. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
  38. Self-supervised label augmentation via input transformations. In ICML, 2020.
  39. Deepgcns: Can gcns go as deep as cnns? In ICCV, 2019.
  40. Focal loss for dense object detection. In ICCV, 2017.
  41. An MLP-based algorithm for efficient contrastive graph recommendations. In SIGIR, 2022.
  42. Wiki-cs: A wikipedia-based benchmark for graph neural networks. arXiv, 2020.
  43. Tudataset: A collection of benchmark datasets for learning with graphs. arXiv, 2020.
  44. Muthén, B. Latent variable analysis. The Sage handbook of quantitative methodology for the social sciences, 2004.
  45. graph2vec: Learning distributed representations of graphs. arXiv, 2017.
  46. Pitfalls of graph neural network evaluation. arXiv, 2018.
  47. Weisfeiler-lehman graph kernels. JMLR, 2011.
  48. Normalized cuts and image segmentation. TPAMI, 2000.
  49. Opening the black box of deep neural networks via information. arXiv, 2017.
  50. A simple min-cut algorithm. Journal of the ACM, 1997.
  51. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In ICLR, 2020.
  52. Patient knowledge distillation for bert model compression. In EMNLP, 2019.
  53. All in one: Multi-task prompting for graph neural networks. In KDD, 2023.
  54. Large-scale representation learning on graphs via bootstrapping. In ICLR, 2022.
  55. Contrastive representation distillation. In ICLR, 2020a.
  56. What makes for good views for contrastive learning? NeurIPS, 2020b.
  57. Learning MLPs on graphs: A unified view of effectiveness, robustness, and efficiency. In ICLR, 2023.
  58. Deep learning and the information bottleneck principle. In ITW, 2015.
  59. The information bottleneck method. arXiv, 2000.
  60. Graph attention networks. In ICLR, 2018.
  61. Deep graph infomax. In ICLR, 2019.
  62. Apan: Asynchronous propagation attention network for real-time temporal graph embedding. In SIGMOD, 2021.
  63. Graph explicit neural networks: Explicitly encoding graphs for efficient and accurate inference. In WSDM, 2023.
  64. Anti-money laundering in bitcoin: Experimenting with graph convolutional networks for financial forensics. arXiv, 2019.
  65. Simplifying graph convolutional networks. In ICML, 2019.
  66. Self-supervised representation learning via latent graph prediction. In ICML, 2022.
  67. Infogcl: Information-aware graph contrastive learning. In NeurIPS, 2021.
  68. How powerful are graph neural networks? In ICLR, 2019.
  69. Tinygnn: Learning efficient graph neural networks. In KDD, 2020.
  70. Deep graph kernels. In KDD, 2015.
  71. Extract the knowledge of graph neural networks and go beyond it: An effective knowledge distillation framework. In WWW, 2021.
  72. Graph neural networks are inherently good generalizers: Insights by bridging GNNs and MLPs. In ICLR, 2023a.
  73. Vqgraph: Graph vector-quantization for bridging gnns and mlps. arXiv, 2023b.
  74. Revisiting semi-supervised learning with graph embeddings. In ICML, 2016.
  75. Graph contrastive learning with augmentations. In NeurIPS, 2020.
  76. Graph contrastive learning automated. In ICML, 2021.
  77. Barlow twins: Self-supervised learning via redundancy reduction. In ICML, 2021.
  78. Graphsaint: Graph sampling based inductive learning method. In ICLR, 2020.
  79. Graph-less neural networks: Teaching old MLPs new tricks via distillation. In ICLR, 2022.
  80. Data augmentation for graph neural networks. In AAAI, 2021.
  81. Accelerating large scale real-time gnn inference using channel pruning. In VLDB, 2021.
  82. Deep graph contrastive representation learning. arXiv, 2020.
  83. Graph contrastive learning with adaptive augmentation. In WWW, 2021.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets