Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
157 tokens/sec
GPT-4o
8 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

OpenGraph: Towards Open Graph Foundation Models (2403.01121v4)

Published 2 Mar 2024 in cs.LG, cs.AI, and cs.SI

Abstract: Graph learning has become essential in various domains, including recommendation systems and social network analysis. Graph Neural Networks (GNNs) have emerged as promising techniques for encoding structural information and improving performance in tasks like link prediction and node classification. However, a key challenge remains: the difficulty of generalizing to unseen graph data with different properties. In this work, we propose a novel graph foundation model, called OpenGraph, to address this challenge. Our approach tackles several technical obstacles. Firstly, we enhance data augmentation using a LLM to overcome data scarcity in real-world scenarios. Secondly, we introduce a unified graph tokenizer that enables the model to generalize effectively to diverse graph data, even when encountering unseen properties during training. Thirdly, our developed scalable graph transformer captures node-wise dependencies within the global topological context. Extensive experiments validate the effectiveness of our framework. By adapting OpenGraph to new graph characteristics and comprehending diverse graphs, our approach achieves remarkable zero-shot graph learning performance across various settings. We release the model implementation at https://github.com/HKUDS/OpenGraph.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. Graphllm: Boosting graph reasoning ability of large language model. arXiv preprint arXiv:2310.05845, 2023.
  2. Generative adversarial framework for cold-start item recommendation. In SIGIR, pages 2565–2571, 2022.
  3. Simple and deep graph convolutional networks. In ICML, pages 1725–1735. PMLR, 2020.
  4. Graph unlearning. In SIGSAC, pages 499–513, 2022.
  5. Universal prompt tuning for graph neural networks. NeurIPS, 2023.
  6. Talk like a graph: Encoding graphs for large language models. In NeurIPS 2023 Workshop: New Frontiers in Graph Learning, 2023.
  7. Large-scale learnable graph convolutional networks. In KDD, pages 1416–1424, 2018.
  8. A. E. Gelfand. Gibbs sampling. Journal of the American statistical Association, 95(452):1300–1304, 2000.
  9. Good: A graph out-of-distribution benchmark. NeurIPS, 35:2059–2073, 2022.
  10. Graphedit: Large language models for graph structure learning. arXiv preprint arXiv:2402.15183, 2024.
  11. Lightgcn: Simplifying and powering graph convolution network for recommendation. In SIGIR, pages 639–648, 2020.
  12. Heterogeneous graph transformer. In WWW, pages 2704–2710, 2020.
  13. Universal graph convolutional networks. NeurIPS, 34:10654–10664, 2021.
  14. Automated self-supervised learning for graphs. In ICLR, 2022.
  15. Graph structure learning for robust graph neural networks. In KDD, pages 66–74, 2020.
  16. T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. 2017.
  17. Augmentation-free self-supervised learning on graphs. In AAAI, volume 36, pages 7372–7380, 2022.
  18. Disentangled contrastive learning on graphs. NeurIPS, 34:21872–21884, 2021.
  19. Efficient graph generation with graph recurrent attention networks. NeurIPS, 32, 2019.
  20. Graph self-supervised learning: A survey. Transactions on Knowledge and Data Engineering (TKDE), 35(6):5879–5900, 2022.
  21. Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. In WWW, pages 417–428, 2023.
  22. Are we really making much progress? revisiting, benchmarking and refining heterogeneous graph neural networks. In KDD, pages 1150–1160, 2021.
  23. Disentangled multiplex graph representation learning. In ICML, pages 24983–25005. PMLR, 2023.
  24. Representation learning with large language models for recommendation. arXiv preprint arXiv:2310.15950, 2023.
  25. Graph neural networks for friend ranking in large-scale social platforms. In WWW, pages 2535–2546, 2021.
  26. How powerful is graph convolution for recommendation? In CIKM, pages 1619–1629, 2021.
  27. Gppt: Graph pre-training and prompt tuning to generalize graph neural networks. In KDD, pages 1717–1727, 2022.
  28. All in one: Multi-task prompting for graph neural networks. In KDD, 2023.
  29. Graphgpt: Graph instruction tuning for large language models. arXiv preprint arXiv:2310.13023, 2023.
  30. Graph attention networks. 2018.
  31. Deep graph infomax. In ICLR, 2018.
  32. Traffic flow prediction via spatial temporal graph neural network. In WWW, pages 1082–1092, 2020.
  33. Self-supervised graph learning for recommendation. In SIGIR, pages 726–735, 2021.
  34. Self-supervised learning on graphs: Contrastive, generative, or predictive. Transactions on Knowledge and Data Engineering (TKDE), 2021.
  35. Handling distribution shifts on graphs: An invariance perspective. In ICLR, 2021.
  36. A comprehensive survey on graph neural networks. Transactions on Neural Networks and Learning Systems (TNNLS), 32(1):4–24, 2020.
  37. Automated self-supervised learning for recommendation. In WWW, pages 992–1002, 2023.
  38. Decoupled self-supervised learning for graphs. NeurIPS, 35:620–634, 2022.
  39. How powerful are graph neural networks? In ICLR, 2018.
  40. Graph pre-training and prompt learning for recommendation. arXiv preprint arXiv:2311.16716, 2023.
  41. Graph convolutional neural networks for web-scale recommender systems. In KDD, pages 974–983, 2018.
  42. Graph contrastive learning automated. In ICML, pages 12121–12132. PMLR, 2021.
  43. Graph contrastive learning with augmentations. NeurIPS, 33:5812–5823, 2020.
  44. Xgnn: Towards model-level explanations of graph neural networks. In KDD, pages 430–438, 2020.
  45. Graph transformer networks. NeurIPS, 32, 2019.
  46. Few-shot heterogeneous graph learning via cross-domain knowledge transfer. In KDD, pages 2450–2460, 2022.
  47. Graph attention multi-layer perceptron. In KDD, pages 4560–4570, 2022.
  48. Lorentzian graph convolutional networks. In WWW, pages 1249–1261, 2021.
  49. E. Zheleva and L. Getoor. Preserving the privacy of sensitive relationships in graph data. In International workshop on privacy, security, and trust in KDD, pages 153–171. Springer, 2007.
  50. Instant graph neural networks for dynamic graphs. In KDD, pages 2605–2615, 2022.
  51. Graph contrastive learning with adaptive augmentation. In WWW, pages 2069–2080, 2021.
Citations (19)
View on Semantic Scholar

Summary

  • The paper introduces a unified graph tokenizer to overcome node token set shifts and enhance zero-shot graph learning performance.
  • The paper employs a scalable graph transformer with anchor sampling to efficiently model node dependencies across large graphs.
  • The paper uses LLM-enhanced data augmentation to alleviate data scarcity, significantly improving generalization in diverse domains.

Enhancing Graph Learning Paradigm with OpenGraph: Insights into Zero-Shot Performance and Scalability

Key Contributions and Methodology

The landscape of graph learning tasks, ranging from link prediction in recommender systems to node classification in citation networks, has seen a pivotal shift with the introduction of Graph Neural Networks (GNNs). These networks excel in encoding complex relational data within a graph's structure, thereby setting a new benchmark for performance in numerous applications. Despite their success, GNNs often stumble when faced with generalizing to graph data unseen during training, a significant bottleneck that limits their potential applications.

Addressing this challenge, the paper on the OpenGraph (\model) model presents a landscape-altering approach aimed at transcending these limitations. Through the development of a general graph foundation model, this work stands out by efficiently interpreting complex topological structures across a wide spectrum of graph data, thereby markedly enhancing zero-shot learning capabilities across diverse domains. Key technical challenges tackled in this work include the varying node token set shifts, efficient node-wise dependency modeling, and addressing domain-specific data scarcity.

The \model\ introduces three cardinal innovations to overcome these hurdles:

  1. Unified Graph Tokenizer: A first-of-its-kind technique focusing on the creation of universal graph tokens, effectively addressing the challenge of node token set shifts between different graphs. This tokenizer transforms any input graph into a unified sequence of tokens, embedding the rich topology-aware features with minimal loss of structural information.
  2. Scalable Graph Transformer: Pioneering a graph transformer architecture that leverages efficient self-attention mechanisms with anchor sampling, this model component ensures the scalability of node-wise dependency encoding within the graphs.
  3. LLM-Enhanced Data Augmentation: By integrating LLMs for synthetic graph generation, this method significantly alleviates the issue of domain-specific data scarcity. It enriches the model's training data with diversified and realistic graph scenarios, preparing the model for robust zero-shot learning.

Empirical Evaluation and Findings

Extensive experiments validate the \model's commendable zero-shot learning performance across various task settings and domains. Notably, when compared to established baselines in both one-shot and five-shot learning cases, \model\ showcases superior generalization capabilities, attributed to its advanced graph tokenizer and transformer architecture. Moreover, the paper critically evaluates the impact of different model configurations, revealing insights into the scalability and efficiency of the proposed model.

Implications and Future Directions

The advancements presented in the OpenGraph model have far-reaching implications for the field of graph learning. By effectively bridging the gap between pre-training and generalization across unseen datasets, this work opens avenues for developing more sophisticated and versatile graph foundation models. Moving forward, it is anticipated that further exploration could focus on enhancing the model's interpretability and adaptability, potentially extending to dynamic and temporal graph data for broader applications.

In the grand scheme of AI research, \model\ not only contributes to the progression of graph learning methodologies but also sets a precedent for leveraging the synergy between tokenization, transformer-based architectures, and synthetic data augmentation strategies. As we look to the future, the methodologies refined and introduced by \model\ hold the promise of unlocking new realms of possibilities within and beyond graph-based data analysis.