Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
149 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Learning a Mini-batch Graph Transformer via Two-stage Interaction Augmentation (2407.09904v1)

Published 13 Jul 2024 in cs.LG

Abstract: Mini-batch Graph Transformer (MGT), as an emerging graph learning model, has demonstrated significant advantages in semi-supervised node prediction tasks with improved computational efficiency and enhanced model robustness. However, existing methods for processing local information either rely on sampling or simple aggregation, which respectively result in the loss and squashing of critical neighbor information.Moreover, the limited number of nodes in each mini-batch restricts the model's capacity to capture the global characteristic of the graph. In this paper, we propose LGMformer, a novel MGT model that employs a two-stage augmented interaction strategy, transitioning from local to global perspectives, to address the aforementioned bottlenecks.The local interaction augmentation (LIA) presents a neighbor-target interaction Transformer (NTIformer) to acquire an insightful understanding of the co-interaction patterns between neighbors and the target node, resulting in a locally effective token list that serves as input for the MGT. In contrast, global interaction augmentation (GIA) adopts a cross-attention mechanism to incorporate entire graph prototypes into the target node epresentation, thereby compensating for the global graph information to ensure a more comprehensive perception. To this end, LGMformer achieves the enhancement of node representations under the MGT paradigm.Experimental results related to node classification on the ten benchmark datasets demonstrate the effectiveness of the proposed method. Our code is available at https://github.com/l-wd/LGMformer.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. Scaling graph neural networks with approximate pagerank. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2020.
  2. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI conference on artificial intelligence, 2020.
  3. Nagphormer: A tokenized graph transformer for node classification in large graphs. In International Conference on Learning Representations, 2023a.
  4. Improving expressivity of gnns with subgraph-specific factor embedded normalization. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023b.
  5. Distribution knowledge embedding for graph pooling. IEEE Transactions on Knowledge and Data Engineering, 35(8):7898–7908, 2023c.
  6. Hybrid riemannian graph-embedding metric learning for image set classification. IEEE transactions on big data, 9(1):75–92, 2021.
  7. Adaptive universal generalized pagerank graph neural network. arXiv preprint arXiv:2006.07988, 2020.
  8. V. P. Dwivedi and X. Bresson. A generalization of transformer networks to graphs. arXiv preprint arXiv:2012.09699, 2020.
  9. Grand+: Scalable graph random neural networks. In Proceedings of the ACM Web Conference 2022, 2022.
  10. Vcr-graphormer: A mini-batch graph transformer via virtual connections. In The Twelfth International Conference on Learning Representations, 2023.
  11. Walking with attention: Self-guided walking for heterogeneous graph embedding. IEEE Transactions on Knowledge and Data Engineering, 34(12):6047–6060, 2021.
  12. Heterogeneous graph prototypical networks for few-shot node classification. In International Conference on Neural Information Processing, pages 540–555. Springer, 2023.
  13. Manifold-based multi-graph embedding for semi-supervised classification. Pattern Recognition Letters, 182:53–59, 2024.
  14. Open graph benchmark: Datasets for machine learning on graphs. In Advances in Neural Information Processing Systems, 2020.
  15. Tailoring self-attention for graph via rooted subtrees. In Advances in Neural Information Processing Systems, 2024.
  16. Hierarchical spatio–temporal graph convolutional networks and transformer network for traffic flow forecasting. IEEE Transactions on Intelligent Transportation Systems, 24(4):3855–3867, 2023.
  17. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016.
  18. T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
  19. Goat: A global transformer on large-scale graphs. In International Conference on Machine Learning, 2023.
  20. Rethinking graph transformers with spectral attention. In Advances in Neural Information Processing Systems, 2021.
  21. Graph classification using structural attention. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2018.
  22. Large scale learning on non-homophilous graphs: New benchmarks and strong simple methods. In Advances in Neural Information Processing Systems, 2021.
  23. Transmission interface power flow adjustment: A deep reinforcement learning approach based on multi-task attribution map. IEEE Transactions on Power Systems, 39(2):3324–3335, 2024a.
  24. Progressive dcision-making framework for power system topology control. Expert Systems with Applications, 235:121070, 2024b.
  25. Single-cell biological network inference using a heterogeneous graph transformer. Nature Communications, 14(1):964, 2023a.
  26. Polyformer: Scalable graph transformer via polynomial attention. 2023b.
  27. P. Mernyei and C. Cangea. Wiki-cs: A wikipedia-based benchmark for graph neural networks. arXiv preprint arXiv:2007.02901, 2020.
  28. A critical look at the evaluation of gnns under heterophily: Are we really making progress? arXiv preprint arXiv:2302.11640, 2023.
  29. Recipe for a general, powerful, scalable graph transformer. In Advances in Neural Information Processing Systems, 2022.
  30. Exphormer: Sparse transformers for graphs. In International Conference on Machine Learning, 2023.
  31. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
  32. Graph attention networks. In International Conference on Learning Representations, 2018.
  33. Nodeformer: A scalable graph structure learning transformer for node classification. In Advances in Neural Information Processing Systems, 2022.
  34. DIFFormer: Scalable (graph) transformers induced by energy constrained diffusion. In The Eleventh International Conference on Learning Representations, 2023a.
  35. Sgformer: Simplifying and empowering transformers for large-graph representations. In Advances in Neural Information Processing Systems, 2023b.
  36. Do transformers really perform badly for graph representation? In Advances in Neural Information Processing Systems, 2021.
  37. Graphsaint: Graph sampling based inductive learning method. arXiv preprint arXiv:1907.04931, 2019.
  38. Dynamic graph message passing networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
  39. Hierarchical graph transformer with adaptive node sampling. In Advances in Neural Information Processing Systems, 2022.
  40. Gophormer: Ego-graph transformer for node classification. arXiv preprint arXiv:2110.13094, 2021.
  41. Transition propagation graph neural networks for temporal networks. IEEE Transactions on Neural Networks and Learning Systems, 2022.
  42. Temporal aggregation and propagation graph neural networks for dynamic representation. IEEE Transactions on Knowledge and Data Engineering, 2023.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - l-wd/LGMformer (3 stars)