Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Hi-GMAE: Hierarchical Graph Masked Autoencoders (2405.10642v1)

Published 17 May 2024 in cs.LG

Abstract: Graph Masked Autoencoders (GMAEs) have emerged as a notable self-supervised learning approach for graph-structured data. Existing GMAE models primarily focus on reconstructing node-level information, categorizing them as single-scale GMAEs. This methodology, while effective in certain contexts, tends to overlook the complex hierarchical structures inherent in many real-world graphs. For instance, molecular graphs exhibit a clear hierarchical organization in the form of the atoms-functional groups-molecules structure. Hence, the inability of single-scale GMAE models to incorporate these hierarchical relationships often leads to their inadequate capture of crucial high-level graph information, resulting in a noticeable decline in performance. To address this limitation, we propose Hierarchical Graph Masked AutoEncoders (Hi-GMAE), a novel multi-scale GMAE framework designed to handle the hierarchical structures within graphs. First, Hi-GMAE constructs a multi-scale graph hierarchy through graph pooling, enabling the exploration of graph structures across different granularity levels. To ensure masking uniformity of subgraphs across these scales, we propose a novel coarse-to-fine strategy that initiates masking at the coarsest scale and progressively back-projects the mask to the finer scales. Furthermore, we integrate a gradual recovery strategy with the masking process to mitigate the learning challenges posed by completely masked subgraphs. Diverging from the standard graph neural network (GNN) used in GMAE models, Hi-GMAE modifies its encoder and decoder into hierarchical structures. This entails using GNN at the finer scales for detailed local graph analysis and employing a graph transformer at coarser scales to capture global information. Our experiments on 15 graph datasets consistently demonstrate that Hi-GMAE outperforms 17 state-of-the-art self-supervised competitors.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (59)
  1. Language models are few-shot learners. In NeurIPS.
  2. When to Pre-Train Graph Neural Networks? From Data Generation Perspective!. In SIGKDD.
  3. Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: a library for support vector machines. TIST (2011).
  4. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In ACL.
  5. Graph Neural Networks with Learnable Structural and Positional Representations. In ICLR.
  6. Matthias Fey and Jan E. Lenssen. 2019. Fast Graph Representation Learning with PyTorch Geometric. In ICLR Workshop.
  7. Hongyang Gao and Shuiwang Ji. 2019. Graph u-nets. In ICML.
  8. Hierarchical graph learning for protein–protein interaction. Nature Communications (2023).
  9. Semi-Implicit Graph Variational Auto-Encoders. In NeurIPS.
  10. GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learner. In WWW.
  11. GraphMAE: Self-Supervised Masked Graph Autoencoders. In SIGKDD.
  12. Strategies for Pre-training Graph Neural Networks. In ICLR.
  13. HGCVAE: Integrating Generative and Contrastive Learning for Heterogeneous Graph Learning. arXiv:2310.11102 (2023).
  14. Contrastive pre-training of GNNs on heterogeneous graphs. In CIKM. 803–812.
  15. George Karypis and Vipin Kumar. 1997. METIS: A software package for partitioning unstructured graphs, partitioning meshes, and computing fill-reducing orderings of sparse matrices. (1997).
  16. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv:1412.6980 (2014).
  17. Thomas N Kipf and Max Welling. 2016. Variational graph auto-encoders. arXiv:1611.07308 (2016).
  18. Self-attention graph pooling. In ICML.
  19. KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction. In SIGKDD. 857–867.
  20. What’s Behind the Mask: Understanding Masked Graph Modeling for Graph Autoencoders. In SIGKDD.
  21. Let invariant rationale discovery inspire graph contrastive learning. In ICLR.
  22. Where to Mask: Structure-Guided Masking for Graph Masked Autoencoders. In IJCAI.
  23. Gapformer: Graph Transformer with Graph Pooling for Node Classification. In IJCAI.
  24. Graph Pooling for Graph Neural Networks: Progress, Challenges, and Opportunities. In IJCAI.
  25. Mask and Reason: Pre-Training Knowledge Graph Transformers for Complex Logical Queries. In SIGKDD.
  26. Rethinking Tokenizer and Decoder in Masked Graph Modeling for Molecules. NeurIPS (2023).
  27. Tudataset: A collection of benchmark datasets for learning with graphs. arXiv:2007.08663 (2020).
  28. Adversarially Regularized Graph Autoencoder for Graph Embedding. In IJCAI.
  29. GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training. In SIGKDD.
  30. Recipe for a General, Powerful, Scalable Graph Transformer. In NeurIPS.
  31. GiGaMAE: Generalizable Graph Masked Autoencoder via Collaborative Latent Space Reconstruction. In CIKM.
  32. Teague Sterling and John J Irwin. 2015. ZINC 15–ligand discovery for everyone. Journal of chemical information and modeling (2015).
  33. InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization. In ICLR.
  34. GPPT: Graph Pre-training and Prompt Tuning to Generalize Graph Neural Networks. In SIGKDD.
  35. S2GAE: Self-Supervised Graph Autoencoders Are Generalizable Learners with Graph Masking. In WSDM.
  36. Heterogeneous graph masked autoencoders. In AAAI.
  37. RARE: Robust Masked Graph Autoencoder. IEEE TKDE (2023).
  38. Deep Graph Infomax. In ICLR.
  39. Generative and Contrastive Paradigms Are Complementary for Graph Self-Supervised Learning. arXiv:2310.15523 (2023).
  40. MoleculeNet: a benchmark for molecular machine learning. Chem. Sci. (2018).
  41. SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation. In WWW.
  42. Mole-BERT: Rethinking Pre-training Graph Neural Networks for Molecules. In ICLR.
  43. A survey of pretraining on graphs: Taxonomy, methods, and applications. arXiv:2202.07893 (2022).
  44. InfoGCL: Information-Aware Graph Contrastive Learning. In NeurIPS.
  45. How Powerful are Graph Neural Networks?. In ICLR.
  46. Self-supervised graph-level representation learning with local and global structure. In ICML.
  47. Skeletonmae: graph-based masked autoencoder for skeleton sequence pre-training. In ICCV.
  48. Do Transformers Really Perform Badly for Graph Representation?. In NeurIPS.
  49. Hierarchical Graph Representation Learning with Differentiable Pooling. In NeurIPS.
  50. Graph contrastive learning automated. In ICML.
  51. Graph Contrastive Learning with Augmentations. In NeurIPS.
  52. ProteinMAE: Masked Autoencoder for Protein Surface Self-supervised Learning. Bioinformatics (2023).
  53. Hierarchical molecular graph self-supervised learning for property prediction. Communications Chemistry (2023).
  54. Contrastive Cross-scale Graph Knowledge Synergy. In SIGKDD.
  55. COSTA: Covariance-Preserving Feature Augmentation for Graph Contrastive Learning. In SIGKDD.
  56. Hierarchical Graph Transformer with Adaptive Node Sampling. In NeurIPS.
  57. Masked Graph Autoencoder with Non-discrete Bandwidths. In WWW.
  58. Unified 2d and 3d pre-training of molecular representations. In SIGKDD.
  59. Hierarchical Transformer for Scalable Graph Learning. In IJCAI.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets