FALCON: Feature-Label Constrained Graph Net Collapse for Memory Efficient GNNs (2312.16542v1)
Abstract: Graph Neural Network (GNN) ushered in a new era of machine learning with interconnected datasets. While traditional neural networks can only be trained on independent samples, GNN allows for the inclusion of inter-sample interactions in the training process. This gain, however, incurs additional memory cost, rendering most GNNs unscalable for real-world applications involving vast and complicated networks with tens of millions of nodes (e.g., social circles, web graphs, and brain graphs). This means that storing the graph in the main memory can be difficult, let alone training the GNN model with significantly less GPU memory. While much of the recent literature has focused on either mini-batching GNN methods or quantization, graph reduction methods remain largely scarce. Furthermore, present graph reduction approaches have several drawbacks. First, most graph reduction focuses only on the inference stage (e.g., condensation and distillation) and requires full graph GNN training, which does not reduce training memory footprint. Second, many methods focus solely on the graph's structural aspect, ignoring the initial population feature-label distribution, resulting in a skewed post-reduction label distribution. Here, we propose a Feature-Label COnstrained graph Net collapse, FALCON, to address these limitations. Our three core contributions lie in (i) designing FALCON, a topology-aware graph reduction technique that preserves feature-label distribution; (ii) implementation of FALCON with other memory reduction methods (i.e., mini-batched GNN and quantization) for further memory reduction; (iii) extensive benchmarking and ablation studies against SOTA methods to evaluate FALCON memory reduction. Our extensive results show that FALCON can significantly collapse various public datasets while achieving equal prediction quality across GNN models. Code: https://github.com/basiralab/FALCON
- A. Bessadok, M. A. Mahjoub, and I. Rekik, “Graph neural networks in network neuroscience,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
- C. Gao, Y. Zheng, N. Li, Y. Li, Y. Qin, J. Piao, Y. Quan, J. Chang, D. Jin, X. He, and Y. Li, “A survey of graph neural networks for recommender systems: Challenges, methods, and directions,” ACM Transactions on Recommender Systems, vol. 1, no. 1, pp. 1–51, Mar 31, 2023.
- Q. Wu, Y. Chen, C. Yang, and J. Yan, “Energy-based out-of-distribution detection for graph neural networks,” in International Conference on Learning Representations (ICLR), 2023.
- Z. Wang, V. N. Ioannidis, H. Rangwala, T. Arai, R. Brand, M. Li, and Y. Nakayama, “Graph neural networks in life sciences: Opportunities and solutions,” in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, ser. KDD ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 4834–4835. [Online]. Available: https://doi.org/10.1145/3534678.3542628
- T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in International Conference on Learning Representations (ICLR), 2017.
- P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” in International Conference on Learning Representations (ICLR), 2018.
- J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl, “Message passing neural networks,” Machine Learning Meets Quantum Physics, p. 199, 2020.
- W.-L. Chiang, X. Liu, S. Si, Y. Li, S. Bengio, and C.-J. Hsieh, “Cluster-gcn: An efficient algorithm for training deep and large graph convolutional networks,” in ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2019. [Online]. Available: https://arxiv.org/pdf/1905.07953.pdf
- G. Karypis and V. Kumar, “Kumar, v.: A fast and high quality multilevel scheme for partitioning irregular graphs. siam journal on scientific computing 20(1), 359-392,” Siam Journal on Scientific Computing, vol. 20, 01 1999.
- M. Fey, J. E. Lenssen, F. Weichert, and J. Leskovec, “GNNAutoScale: Scalable and expressive graph neural networks via historical embeddings,” in International Conference on Machine Learning (ICML), 2021.
- Z. Shi, X. Liang, and J. Wang, “LMC: Fast training of GNNs via subgraph sampling with provable convergence,” in International Conference on Learning Representations (ICLR), 2023. [Online]. Available: https://openreview.net/forum?id=5VBBA91N6n
- F. Wu, A. Souza, T. Zhang, C. Fifty, T. Yu, and K. Weinberger, “Simplifying graph convolutional networks,” in Proceedings of the 36th International Conference on Machine Learning (ICML). PMLR, 2019, pp. 6861–6871.
- F. Frasca, E. Rossi, D. Eynard, B. Chamberlain, M. Bronstein, and F. Monti, “Sign: Scalable inception graph neural networks,” in ICML 2020 Workshop on Graph Representation Learning and Beyond, 2020.
- Z. Liu, K. Zhou, F. Yang, L. Li, R. Chen, and X. Hu, “EXACT: Scalable graph neural networks training via extreme activation compression,” in International Conference on Learning Representations (ICLR), 2022. [Online]. Available: https://openreview.net/forum?id=vkaMaq95_rX
- J. Chen, L. Zheng, Z. Yao, D. Wang, I. Stoica, M. W. Mahoney, and J. E. Gonzalez, “Actnn: Reducing training memory footprint via 2-bit activation compressed training,” in International Conference on Machine Learning (ICML), 2021.
- P. Prochazka, M. Mares, and M. Dedic, “Scalable graph size reduction for efficient gnn application,” in Information technologies – Applications and Theory, 2022.
- Z. Huang, S. Zhang, C. Xi, T. Liu, and M. Zhou, “Scaling up graph neural networks via graph coarsening,” in In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’21), 2021.
- C. Adnel and I. Rekik, “Affordable graph neural network framework using topological graph contraction,” Medical Image Learning with Limited and Noisy Data. Springer Nature Switzerland, pp. 35–46, 2023.
- W. Jin, L. Zhao, S. Zhang, Y. Liu, J. Tang, and N. Shah, “Graph condensation for graph neural networks,” in International Conference on Learning Representations (ICLR), 2022. [Online]. Available: https://openreview.net/forum?id=WLEx3Jo4QaB
- W. Jin, X. Tang, H. Jiang, Z. Li, D. Zhang, J. Tang, and B. Yin, “Condensing graphs via one-step gradient matching,” in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022, pp. 720–730.
- C. Liu, X. Ma, Y. Zhan, L. Ding, D. Tao, B. Du, W. Hu, and D. Mandic, “Comprehensive graph gradual pruning for sparse training in graph neural networks,” IEEE Transactions on Neural Networks and Learning Systems, 2022.
- U. Brandes, “A faster algorithm for betweenness centrality,” The Journal of Mathematical Sociology, vol. 25, 03 2004.
- S. Wandelt, X. Shi, and X. Sun, “Complex network metrics: Can deep learning keep up with tailor-made reference algorithms?” IEEE Access, vol. 8, pp. 68 114–68 123, 2020.
- L. Page, S. Brin, R. Motwani, and T. Winograd, “The pagerank citation ranking : Bringing order to the web,” in The Web Conference, 1999.
- M. Zitnik and J. Leskovec, “Predicting multicellular function through multi-layer tissue networks,” Bioinformatics, vol. 33, no. 14, pp. i190–i198, 07 2017.
- H. Zeng, H. Zhou, A. Srivastava, R. Kannan, and V. Prasanna, “GraphSAINT: Graph sampling based inductive learning method,” in International Conference on Learning Representations, 2020. [Online]. Available: https://openreview.net/forum?id=BJe8pkHFwS
- ——, “Accurate, efficient and scalable graph embedding,” in 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS), May 2019.
- J. Yang, R. Shi, D. Wei, Z. Liu, L. Zhao, B. Ke, H. Pfister, and B. Ni, “Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification,” Scientific Data, vol. 10, no. 1, p. 41, 2023.
- Z. Yang, W. W. Cohen, and R. Salakhutdinov, “Revisiting semi-supervised learning with graph embeddings,” in Proceedings of the 36th International Conference on Machine Learning (ICML), 2016.