Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

PUMA: Efficient Continual Graph Learning for Node Classification with Graph Condensation (2312.14439v2)

Published 22 Dec 2023 in cs.LG

Abstract: When handling streaming graphs, existing graph representation learning models encounter a catastrophic forgetting problem, where previously learned knowledge of these models is easily overwritten when learning with newly incoming graphs. In response, Continual Graph Learning (CGL) emerges as a novel paradigm enabling graph representation learning from streaming graphs. Our prior work, Condense and Train (CaT) is a replay-based CGL framework with a balanced continual learning procedure, which designs a small yet effective memory bankn for replaying. Although the CaT alleviates the catastrophic forgetting problem, there exist three issues: (1) The graph condensation only focuses on labelled nodes while neglecting abundant information carried by unlabelled nodes; (2) The continual training scheme of the CaT overemphasises on the previously learned knowledge, limiting the model capacity to learn from newly added memories; (3) Both the condensation process and replaying process of the CaT are time-consuming. In this paper, we propose a PsUdo-label guided Memory bAnk (PUMA) CGL framework, extending from the CaT to enhance its efficiency and effectiveness by overcoming the above-mentioned weaknesses and limits. To fully exploit the information in a graph, PUMA expands the coverage of nodes during graph condensation with both labelled and unlabelled nodes. Furthermore, a training-from-scratch strategy is proposed to upgrade the previous continual learning scheme for a balanced training between the historical and the new graphs. Besides, PUMA uses a one-time prorogation and wide graph encoders to accelerate the graph condensation and the graph encoding process in the training stage to improve the efficiency of the whole framework. Extensive experiments on six datasets for the node classification task demonstrate the state-of-the-art performance and efficiency over existing methods.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. Y. Liu, R. Qiu, and Z. Huang, “Cat: Balanced continual graph learning with graph condensation,” in ICDM, 2023.
  2. Q. Guo, F. Zhuang, C. Qin, H. Zhu, X. Xie, H. Xiong, and Q. He, “A survey on knowledge graph-based recommender systems,” TKDE, 2022.
  3. P. Xie, M. Ma, T. Li, S. Ji, S. Du, Z. Yu, and J. Zhang, “Spatio-temporal dynamic graph relation learning for urban metro flow prediction,” TKDE, 2023.
  4. L. Zhang, Y. Jiang, and Y. Yang, “Gnngo3d: Protein function prediction based on 3d structure and functional hierarchy learning,” TKDE, 2023.
  5. Y. Xu, Y. Zhang, W. Guo, H. Guo, R. Tang, and M. Coates, “Graphsail: Graph structure aware incremental learning for recommender systems,” in CIKM, 2020.
  6. A. A. Daruna, M. Gupta, M. Sridharan, and S. Chernova, “Continual learning of knowledge graph embeddings,” IEEE Robotics Autom. Lett., 2021.
  7. R. Qiu, H. Yin, Z. Huang, and T. Chen, “GAG: global attributed graph neural network for streaming session-based recommendation,” in SIGIR, 2020.
  8. F. Zhou and C. Cao, “Overcoming catastrophic forgetting in graph neural networks with experience replay,” in AAAI, 2021.
  9. H. Liu, Y. Yang, and X. Wang, “Overcoming catastrophic forgetting in graph neural networks,” in AAAI, 2021.
  10. X. Zhang, D. Song, and D. Tao, “Sparsified subgraph memory for continual graph representation learning,” in ICDM, 2022.
  11. X. Zhang, D. Song, and D. Tao, “Hierarchical prototype networks for continual graph representation learning,” TPAMI, 2023.
  12. R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann, “Shortcut learning in deep neural networks,” Nature Machine Intelligence, 2020.
  13. K. Xu, W. Hu, J. Leskovec, and S. Jegelka, “How powerful are graph neural networks?” in ICLR, 2019.
  14. Y. Li, D. Tarlow, M. Brockschmidt, and R. S. Zemel, “Gated graph sequence neural networks,” in ICLR, 2016.
  15. Y. Tang, R. Qiu, Y. Liu, X. Li, and Z. Huang, “Casegnn: Graph neural networks for legal case retrieval with text-attributed graphs,” ECIR, 2024.
  16. R. Qiu, Z. Huang, J. Li, and H. Yin, “Exploiting cross-session information for session-based recommendation with graph neural networks,” TOIS, 2020.
  17. T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in ICLR, 2017.
  18. P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” in ICLR, 2018.
  19. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in NeurIPS, 2017.
  20. F. Wu, A. H. S. Jr., T. Zhang, C. Fifty, T. Yu, and K. Q. Weinberger, “Simplifying graph convolutional networks,” in ICML, 2019.
  21. W. L. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” in NeurIPS, 2017.
  22. J. Kirkpatrick, R. Pascanu, N. C. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell, “Overcoming catastrophic forgetting in neural networks,” CoRR, vol. abs/1612.00796, 2016.
  23. R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars, “Memory aware synapses: Learning what (not) to forget,” in ECCV, 2018.
  24. D. Lopez-Paz and M. Ranzato, “Gradient episodic memory for continual learning,” in NeurIPS, 2017.
  25. Z. Li and D. Hoiem, “Learning without forgetting,” TPAMI, 2018.
  26. X. Zhang, D. Song, and D. Tao, “Ricci curvature-based graph sparsification for continual graph representation learning,” TNNLS, 2023.
  27. J. Su, D. Zou, Z. Zhang, and C. Wu, “Towards robust graph incremental learning on evolving graphs,” in ICML, 2023.
  28. X. Zhang, D. Song, and D. Tao, “Cglb: Benchmark tasks for continual graph learning,” in NeurIPS Systems Datasets and Benchmarks Track, 2022.
  29. J. Ko, S. Kang, and K. Shin, “Begin: Extensive benchmark scenarios and an easy-to-use framework for graph continual learning,” CoRR, vol. abs/2211.14568, 2022.
  30. T. Wang, J. Zhu, A. Torralba, and A. A. Efros, “Dataset distillation,” CoRR, vol. abs/1811.10959, 2018.
  31. B. Zhao, K. R. Mopuri, and H. Bilen, “Dataset condensation with gradient matching,” in ICLR, 2021.
  32. B. Zhao and H. Bilen, “Dataset condensation with differentiable siamese augmentation,” in ICML, 2021.
  33. K. Wang, B. Zhao, X. Peng, Z. Zhu, S. Yang, S. Wang, G. Huang, H. Bilen, X. Wang, and Y. You, “CAFE: learning to condense dataset by aligning features,” in CVPR, 2022.
  34. B. Zhao and H. Bilen, “Dataset condensation with distribution matching,” in WACV, 2023.
  35. G. Cazenavette, T. Wang, A. Torralba, A. A. Efros, and J. Zhu, “Generalizing dataset distillation via deep generative prior,” in CVPR, 2023.
  36. W. Jin, L. Zhao, S. Zhang, Y. Liu, J. Tang, and N. Shah, “Graph condensation for graph neural networks,” in ICLR, 2022.
  37. W. Jin, X. Tang, H. Jiang, Z. Li, D. Zhang, J. Tang, and B. Yin, “Condensing graphs via one-step gradient matching,” in KDD, 2022.
  38. X. Gao, T. Chen, Y. Zang, W. Zhang, Q. V. H. Nguyen, K. Zheng, and H. Yin, “Graph condensation for inductive node representation learning,” ICDE, 2024.
  39. Y. Chen, M. Welling, and A. J. Smola, “Super-samples from kernel herding,” in UAI, 2010.
  40. Z. Wang and J. Ye, “Querying discriminative and representative samples for batch mode active learning,” in KDD, 2013.
  41. O. Sener and S. Savarese, “Active learning for convolutional neural networks: A core-set approach,” in ICLR, 2018.
  42. M. Liu, S. Li, X. Chen, and L. Song, “Graph condensation via receptive field distribution matching,” CoRR, vol. abs/2206.13697, 2022.
  43. X. Zheng, M. Zhang, C. Chen, Q. V. H. Nguyen, X. Zhu, and S. Pan, “Structure-free graph condensation: From large-scale graphs to condensed graph-free data,” NeurIPS, 2023.
  44. B. Yang, K. Wang, Q. Sun, C. Ji, X. Fu, H. Tang, Y. You, and J. Li, “Does graph distillation see like vision dataset counterpart?” NeurIPS, 2024.
  45. W. Masarczyk and I. Tautkute, “Reducing catastrophic forgetting with learning on synthetic data,” in CVPR Workshop, 2020.
  46. F. Wiewel and B. Yang, “Condensed composite memory continual learning,” in IJCNN, 2021.
  47. A. Rosasco, A. Carta, A. Cossu, V. Lomonaco, and D. Bacciu, “Distilled replay: Overcoming forgetting through synthetic samples,” in CSSL Workshop, 2021.
  48. M. Sangermano, A. Carta, A. Cossu, and D. Bacciu, “Sample condensation in online continual learning,” in IJCNN, 2022.
  49. J. Gu, K. Wang, W. Jiang, and Y. You, “Summarizing stream data for memory-restricted online continual learning,” CoRR, vol. abs/2305.16645, 2023.
  50. H. Han, X. Liu, H. Mao, M. Torkamani, F. Shi, V. Lee, and J. Tang, “Alternately optimized graph neural networks,” in ICML, 2023.
  51. D.-H. Lee et al., “Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks,” in ICML Workshop, 2013.
  52. A. McCallum, K. Nigam, J. Rennie, and K. Seymore, “Automating the construction of internet portals with machine learning,” Inf. Retr., 2000.
  53. W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec, “Open graph benchmark: Datasets for machine learning on graphs,” in NeurIPS, 2020.
  54. D. Zhou, Q. Wang, Z. Qi, H. Ye, D. Zhan, and Z. Liu, “Deep class-incremental learning: A survey,” CoRR, vol. abs/2302.03648, 2023.
  55. L. Wang, X. Zhang, H. Su, and J. Zhu, “A comprehensive survey of continual learning: Theory, method and application,” CoRR, vol. abs/2302.00487, 2023.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now