Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Size Generalization of Graph Neural Networks on Biological Data: Insights and Practices from the Spectral Perspective (2305.15611v4)

Published 24 May 2023 in cs.LG and cs.AI

Abstract: We investigate size-induced distribution shifts in graphs and assess their impact on the ability of graph neural networks (GNNs) to generalize to larger graphs relative to the training data. Existing literature presents conflicting conclusions on GNNs' size generalizability, primarily due to disparities in application domains and underlying assumptions concerning size-induced distribution shifts. Motivated by this, we take a data-driven approach: we focus on real biological datasets and seek to characterize the types of size-induced distribution shifts. Diverging from prior approaches, we adopt a spectral perspective and identify that spectrum differences induced by size are related to differences in subgraph patterns (e.g., average cycle lengths). While previous studies have identified that the inability of GNNs in capturing subgraph information negatively impacts their in-distribution generalization, our findings further show that this decline is more pronounced when evaluating on larger test graphs not encountered during training. Based on these spectral insights, we introduce a simple yet effective model-agnostic strategy, which makes GNNs aware of these important subgraph patterns to enhance their size generalizability. Our empirical results reveal that our proposed size-insensitive attention strategy substantially enhances graph classification performance on large test graphs, which are 2-10 times larger than the training graphs, resulting in an improvement in F1 scores by up to 8%.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (40)
  1. Analyzing the expressive power of graph neural networks in a spectral perspective. In Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
  2. Breaking the limits of message passing graph neural networks. In International Conference on Machine Learning, pp.  599–608. PMLR, 2021b.
  3. A scalable approach to size-independent network similarity. ArXiv Preprint ArXiv: 12092684, 2012.
  4. Size-invariant graph representations for graph classification extrapolations. In International Conference on Machine Learning, pp.  837–851. PMLR, 2021.
  5. Beyond low-frequency information in graph convolutional networks. arXiv preprint arXiv:2101.00797, 2021.
  6. Sizeshiftreg: a regularization method for improving size-generalization in graph neural networks. In Advances in Neural Information Processing Systems, 2022.
  7. Learning causally invariant representations for out-of-distribution generalization on graphs. Advances in Neural Information Processing Systems, 35:22131–22148, 2022.
  8. Can graph neural networks count substructures? Advances in neural information processing systems, 33:10383–10395, 2020.
  9. Wasserstein barycenter matching for graph size generalization of message passing neural networks. In International Conference on Machine Learning, pp.  6158–6184. PMLR, 2023.
  10. Convolutional neural networks on graphs with fast localized spectral filtering. In NeurIPS, 2016.
  11. Neural message passing for quantum chemistry. In International conference on machine learning, pp.  1263–1272. PMLR, 2017.
  12. Inductive representation learning on large graphs. In NeurIPS, 2017.
  13. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00687, 2020a.
  14. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118–22133, 2020b.
  15. Strategies for pre-training graph neural networks. In International Conference on Learning Representations (ICLR), 2020c.
  16. Drugood: Out-of-distribution (ood) dataset curator and benchmark for ai-aided drug discovery–a focus on affinity prediction problems with noise annotations. arXiv preprint arXiv:2201.09637, 2022.
  17. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
  18. Understanding attention and generalization in graph neural networks. Advances in neural information processing systems, 32, 2019.
  19. Graph classification using structural attention. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp.  1666–1674, 2018.
  20. Cayleynets: Graph convolutional neural networks with complex rational spectral filters. IEEE Transactions on Signal Processing, 67(1):97–109, 2018.
  21. Transferability of spectral graph convolutional neural networks. The Journal of Machine Learning Research, 22(1):12462–12520, 2021.
  22. Tudataset: A collection of benchmark datasets for learning with graphs. In ICML 2020 Workshop on Graph Representation Learning and Beyond (GRL+ 2020), 2020. URL www.graphlearning.io.
  23. Relational pooling for graph representations. In International Conference on Machine Learning, pp.  4663–4673. PMLR, 2019.
  24. Paton, K. An algorithm for finding a fundamental set of cycles of a graph. Communications of the ACM, 12(9):514–518, 1969.
  25. Learning to simulate complex physics with graph networks. In International conference on machine learning, pp.  8459–8468. PMLR, 2020.
  26. Sysło, M. M. On cycle bases of a graph. Networks, 9(2):123–132, 1979.
  27. Graph Attention Networks. International Conference on Learning Representations (ICLR), 2018. URL https://openreview.net/forum?id=rJXMpikCZ.
  28. Neural execution of graph algorithms. In International Conference on Learning Representations, 2020.
  29. Building powerful and equivariant graph neural networks with structural message-passing. Advances in neural information processing systems, 33:14143–14155, 2020.
  30. Villani, C. et al. Optimal transport: old and new, volume 338. Springer, 2009.
  31. Comparison of descriptor spaces for chemical compound retrieval and classification. In Sixth International Conference on Data Mining (ICDM’06), pp.  678–689, 2006. doi: 10.1109/ICDM.2006.39.
  32. Moleculenet: a benchmark for molecular machine learning. Chemical science, 9(2):513–530, 2018.
  33. How powerful are graph neural networks? International Conference on Learning Representations, 2018.
  34. How neural networks extrapolate: from feedforward to graph neural networks. In International Conference on Learning Representations (ICLR), 2021.
  35. Groupinn: Grouping-based interpretable neural network for classification of limited, noisy brain data. In proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp.  772–782, 2019.
  36. Neural execution engines: Learning to execute subroutines. Advances in Neural Information Processing Systems, 33:17298–17308, 2020.
  37. From local structures to size generalization in graph neural networks. In International Conference on Machine Learning, pp.  11975–11986. PMLR, 2021.
  38. Hierarchical graph representation learning with differentiable pooling. In NeurIPS, pp.  4800–4810, 2018.
  39. An end-to-end deep learning architecture for graph classification. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  40. Ood link prediction generalization capabilities of message-passing gnns in larger test graphs. Advances in Neural Information Processing Systems, 35:20257–20272, 2022.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now