Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
166 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

GLISP: A Scalable GNN Learning System by Exploiting Inherent Structural Properties of Graphs (2401.03114v1)

Published 6 Jan 2024 in cs.LG

Abstract: As a powerful tool for modeling graph data, Graph Neural Networks (GNNs) have received increasing attention in both academia and industry. Nevertheless, it is notoriously difficult to deploy GNNs on industrial scale graphs, due to their huge data size and complex topological structures. In this paper, we propose GLISP, a sampling based GNN learning system for industrial scale graphs. By exploiting the inherent structural properties of graphs, such as power law distribution and data locality, GLISP addresses the scalability and performance issues that arise at different stages of the graph learning process. GLISP consists of three core components: graph partitioner, graph sampling service and graph inference engine. The graph partitioner adopts the proposed vertex-cut graph partitioning algorithm AdaDNE to produce balanced partitioning for power law graphs, which is essential for sampling based GNN systems. The graph sampling service employs a load balancing design that allows the one hop sampling request of high degree vertices to be handled by multiple servers. In conjunction with the memory efficient data structure, the efficiency and scalability are effectively improved. The graph inference engine splits the $K$-layer GNN into $K$ slices and caches the vertex embeddings produced by each slice in the data locality aware hybrid caching system for reuse, thus completely eliminating redundant computation caused by the data dependency of graph. Extensive experiments show that GLISP achieves up to $6.53\times$ and $70.77\times$ speedups over existing GNN systems for training and inference tasks, respectively, and can scale to the graph with over 10 billion vertices and 40 billion edges with limited resources.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (65)
  1. Q. Wang, Z. Mao, B. Wang, and L. Guo, “Knowledge graph embedding: A survey of approaches and applications,” IEEE Transactions on Knowledge and Data Engineering, vol. 29, no. 12, pp. 2724–2743, 2017.
  2. D. Donato, L. Laura, S. Leonardi, and S. Millozzi, “Large scale properties of the webgraph,” The European Physical Journal B, vol. 38, pp. 239–243, 2004.
  3. R. Ying, R. He, K. Chen, P. Eksombatchai, W. L. Hamilton, and J. Leskovec, “Graph convolutional neural networks for web-scale recommender systems,” in Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, 2018, pp. 974–983.
  4. T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” arXiv preprint arXiv:1609.02907, 2016.
  5. W. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” Advances in neural information processing systems, vol. 30, 2017.
  6. Z. Liu, C. Chen, L. Li, J. Zhou, X. Li, L. Song, and Y. Qi, “Geniepath: Graph neural networks with adaptive receptive paths,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, 2019, pp. 4424–4431.
  7. D. I. Shuman, S. K. Narang, P. Frossard, A. Ortega, and P. Vandergheynst, “The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains,” IEEE signal processing magazine, vol. 30, no. 3, pp. 83–98, 2013.
  8. P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” in International Conference on Learning Representations, 2018.
  9. J. Zhou, G. Cui, S. Hu, Z. Zhang, C. Yang, Z. Liu, L. Wang, C. Li, and M. Sun, “Graph neural networks: A review of methods and applications,” AI Open, vol. 1, pp. 57–81, 2020.
  10. J. W. H. Z. Z. Z. L. Lee, “Billion-scale commodity embedding for e-commerce recommendation in alibaba,” SIGKDD explorations, no. Udisk, 2018.
  11. H. Zeng, H. Zhou, A. Srivastava, R. Kannan, and V. Prasanna, “Graphsaint: Graph sampling based inductive learning method,” in International Conference on Learning Representations, 2019.
  12. W. L. Chiang, X. Liu, S. Si, Y. Li, and C. J. Hsieh, “Cluster-gcn: An efficient algorithm for training deep and large graph convolutional networks,” ACM, 2019.
  13. J. Qiu, Q. Chen, Y. Dong, J. Zhang, H. Yang, M. Ding, K. Wang, and J. Tang, “Gcc: Graph contrastive coding for graph neural network pre-training,” in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020, pp. 1150–1160.
  14. M. Wang, D. Zheng, Z. Ye, Q. Gan, M. Li, X. Song, J. Zhou, C. Ma, L. Yu, Y. Gai et al., “Deep graph library: A graph-centric, highly-performant package for graph neural networks,” arXiv preprint arXiv:1909.01315, 2019.
  15. M. Fey and J. E. Lenssen, “Fast graph representation learning with pytorch geometric,” arXiv preprint arXiv:1903.02428, 2019.
  16. D. Zheng, C. Ma, M. Wang, J. Zhou, Q. Su, X. Song, Q. Gan, Z. Zhang, and G. Karypis, “Distdgl: distributed graph neural network training for billion-scale graphs,” in 2020 IEEE/ACM 10th Workshop on Irregular Applications: Architectures and Algorithms (IA3).   IEEE, 2020, pp. 36–44.
  17. D. Zheng, X. Song, C. Yang, D. LaSalle, and G. Karypis, “Distributed hybrid cpu and gpu training for graph neural networks on billion-scale heterogeneous graphs,” in Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022, pp. 4582–4591.
  18. R. Zhu, K. Zhao, H. Yang, W. Lin, C. Zhou, B. Ai, Y. Li, and J. Zhou, “Aligraph: A comprehensive graph neural network platform,” Proceedings of the VLDB Endowment, vol. 12, no. 12.
  19. “Euler github,” Alibaba. [Online]. Available: https://github.com/alibaba/euler
  20. T. Liu, Y. Chen, D. Li, C. Wu, Y. Zhu, J. He, Y. Peng, H. Chen, H. Chen, and C. Guo, “BGL: GPU-Efficient GNN training by optimizing graph data I/O and preprocessing,” in 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23).   Boston, MA: USENIX Association, Apr. 2023, pp. 103–118. [Online]. Available: https://www.usenix.org/conference/nsdi23/presentation/liu-tianfeng
  21. C. Zheng, H. Chen, Y. Cheng, Z. Song, Y. Wu, C. Li, J. Cheng, H. Yang, and S. Zhang, “Bytegnn: efficient graph neural network training at large scale,” Proceedings of the VLDB Endowment, vol. 15, no. 6, pp. 1228–1242, 2022.
  22. G. Li, C. Xiong, A. Thabet, and B. Ghanem, “Deepergcn: All you need to train deeper gcns,” arXiv preprint arXiv:2006.07739, 2020.
  23. Q. Li, Z. Han, and X.-M. Wu, “Deeper insights into graph convolutional networks for semi-supervised learning,” in Proceedings of the AAAI conference on artificial intelligence, vol. 32, no. 1, 2018.
  24. D. Chen, Y. Lin, W. Li, P. Li, J. Zhou, and X. Sun, “Measuring and relieving the over-smoothing problem for graph neural networks from the topological view,” in Proceedings of the AAAI conference on artificial intelligence, vol. 34, no. 04, 2020, pp. 3438–3445.
  25. K. Oono and T. Suzuki, “Graph neural networks exponentially lose expressive power for node classification,” 2019.
  26. C. Cai and Y. Wang, “A note on over-smoothing for graph neural networks,” 2020.
  27. G. Karypis and V. Kumar, “A fast and high quality multilevel scheme for partitioning irregular graphs,” SIAM Journal on scientific Computing, vol. 20, no. 1, pp. 359–392, 1998.
  28. ——, “A parallel algorithm for multilevel graph partitioning and sparse matrix ordering,” Journal of parallel and distributed computing, vol. 48, no. 1, pp. 71–95, 1998.
  29. G. M. Slota, S. Rajamanickam, K. Devine, and K. Madduri, “Partitioning trillion-edge graphs in minutes,” in 2017 IEEE International Parallel and Distributed Processing Symposium (IPDPS).   IEEE, 2017, pp. 646–655.
  30. M. Hanai, T. Suzumura, W. J. Tan, E. Liu, G. Theodoropoulos, and W. Cai, “Distributed edge partitioning for trillion-edge graphs,” Proceedings of the VLDB Endowment, vol. 12, no. 13.
  31. D. Kong, X. Xie, and Z. Zhang, “Clustering-based partitioning for large web graphs,” arXiv preprint arXiv:2201.00472, 2022.
  32. Z. Cai, Q. Zhou, X. Yan, D. Zheng, X. Song, C. Zheng, J. Cheng, and G. Karypis, “Dsp: Efficient gnn training with multiple gpus,” in Proceedings of the 28th ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming, 2023, pp. 392–404.
  33. R. Albert, H. Jeong, and A. L. Barabasi, “Error and attack tolerance of complex networks,” Nature, vol. 406, no. 6794, pp. 378–382, 2000.
  34. D. Donato, L. Laura, S. Leonardi, and S. Millozzi, “Large scale properties of the webgraph,” The European Physical Journal B, vol. 38, no. 2, pp. 239–243, 2004.
  35. J. E. Gonzalez, Y. Low, H. Gu, D. Bickson, and C. Guestrin, “{{\{{PowerGraph}}\}}: Distributed {{\{{Graph-Parallel}}\}} computation on natural graphs,” in 10th USENIX symposium on operating systems design and implementation (OSDI 12), 2012, pp. 17–30.
  36. C. Wan, Y. Li, C. R. Wolfe, A. Kyrillidis, N. S. Kim, and Y. Lin, “Pipegcn: Efficient full-graph training of graph convolutional networks with pipelined feature communication,” 2022.
  37. Z. Jia, S. Lin, M. Gao, M. Zaharia, and A. Aiken, “Improving the accuracy, scalability, and performance of graph neural networks with roc,” Proceedings of Machine Learning and Systems, vol. 2, pp. 187–198, 2020.
  38. X. Wang, H. Ji, C. Shi, B. Wang, P. Cui, P. Yu, and Y. Ye, “Heterogeneous graph attention network,” 2019.
  39. X. Fu, J. Zhang, Z. Meng, and I. King, “Magnn: Metapath aggregated graph neural network for heterogeneous graph embedding,” arXiv preprint arXiv:2002.01680, 2020.
  40. W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec, “Open graph benchmark: Datasets for machine learning on graphs,” Advances in neural information processing systems, vol. 33, pp. 22 118–22 133, 2020.
  41. W. Fan, M. Liu, C. Tian, R. Xu, and J. Zhou, “Incrementalization of graph partitioning algorithms,” Proceedings of the VLDB Endowment, vol. 13, no. 8, pp. 1261–1274, 2020.
  42. K. Andreev and H. Räcke, “Balanced graph partitioning,” ACM, 2004.
  43. G. M. Slota, K. Madduri, and S. Rajamanickam, “Pulp: Scalable multi-objective multi-constraint partitioning for small-world networks,” in IEEE International Conference on Big Data, 2015.
  44. E. Cuthill, “Reducing the bandwidth of sparse symmetric matrices,” in Acm National Conference, 1969.
  45. D. K. Blandford, G. E. Blelloch, and I. A. Kash, “An experimental analysis of a compact graph representation,” Dissertation Abstracts International, 2004.
  46. L. Dhulipala, I. Kabiljo, B. Karrer, G. Ottaviano, S. Pupyrev, and A. Shalita, “Compressing graphs and indexes with recursive graph bisection,” ACM, 2016.
  47. D. Mlakar, M. Winter, M. Parger, and M. Steinberger, “Speculative parallel reverse cuthill-mckee reordering on multi- and many-core architectures,” in 2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS), 2021.
  48. W. Hao, J. X. Yu, C. Lu, and X. Lin, “Speedup graph processing by graph ordering,” in the 2016 International Conference, 2016.
  49. J. Arai, H. Shiokawa, T. Yamamuro, M. Onizuka, and S. Iwamura, “Rabbit order: Just-in-time parallel reordering for fast graph analysis,” in 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS), 2016.
  50. V. Balaji and B. Lucia, “When is graph reordering an optimization? studying the effect of lightweight graph reordering across applications and input graphs,” in 2018 IEEE International Symposium on Workload Characterization (IISWC), 2018.
  51. R. Barik, M. Minutoli, M. Halappanavar, N. R. Tallent, and A. Kalyanaraman, “Vertex reordering for real-world graphs and applications: An empirical evaluation,” in 2020 IEEE International Symposium on Workload Characterization (IISWC), 2020.
  52. P. Boldi and S. Vigna, “The webgraph framework i: Compression techniques,” in International World Wide Web Conference, 2004.
  53. J. S. Vitter, “An efficient algorithm for sequential random sampling,” ACM transactions on mathematical software (TOMS), vol. 13, no. 1, pp. 58–67, 1987.
  54. A. J. Walker, “New fast method for generating discrete random numbers with arbitrary frequency distributions,” Electronics Letters, vol. 8, no. 10, pp. 127–128, 1974.
  55. P. S. Efraimidis and P. G. Spirakis, “Weighted random sampling with a reservoir,” Information processing letters, vol. 97, no. 5, pp. 181–185, 2006.
  56. D. Zhang, X. Huang, Z. Liu, J. Zhou, Z. Hu, X. Song, Z. Ge, L. Wang, Z. Zhang, and Y. Qi, “Agl: A scalable system for industrial-purpose graph machine learning,” Proceedings of the VLDB Endowment, vol. 13, no. 12.
  57. P. Yin, X. Yan, J. Zhou, Q. Fu, Z. Cai, J. Cheng, B. Tang, and M. Wang, “Dgi: An easy and efficient framework for gnn model evaluation,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023, pp. 5439–5450.
  58. Zarr-Python — zarr 2.14.2 documentation. [Online]. Available: https://zarr.readthedocs.io/en/stable/
  59. Blosc main blog page. [Online]. Available: https://www.blosc.org/
  60. Z. Lin, C. Li, Y. Miao, Y. Liu, and Y. Xu, “Pagraph: Scaling gnn training on large graphs via computation-aware caching,” in SoCC ’20: ACM Symposium on Cloud Computing, 2020.
  61. W. Hu, M. Fey, H. Ren, M. Nakata, Y. Dong, and J. Leskovec, “Ogb-lsc: A large-scale challenge for machine learning on graphs,” in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
  62. Twitter-2010. [Online]. Available: https://law.di.unimi.it/webdata/twitter-2010/
  63. Z. Hu, Y. Dong, K. Wang, and Y. Sun, “Heterogeneous graph transformer,” 2020.
  64. A. Tripathy, K. Yelick, and A. Buluç, “Reducing communication in graph neural network training,” in SC20: International Conference for High Performance Computing, Networking, Storage and Analysis.   IEEE, 2020, pp. 1–14.
  65. D. T. T. Van, M. N. Khan, T. H. Afridi, I. Ullah, A. Alam, and Y.-K. Lee, “Gdll: A scalable and share nothing architecture based distributed graph neural networks framework,” IEEE Access, vol. 10, pp. 21 684–21 700, 2022.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets