Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
134 tokens/sec
GPT-4o
10 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Pre-Training and Prompting for Few-Shot Node Classification on Text-Attributed Graphs (2407.15431v1)

Published 22 Jul 2024 in cs.SI, cs.AI, and cs.LG

Abstract: The text-attributed graph (TAG) is one kind of important real-world graph-structured data with each node associated with raw texts. For TAGs, traditional few-shot node classification methods directly conduct training on the pre-processed node features and do not consider the raw texts. The performance is highly dependent on the choice of the feature pre-processing method. In this paper, we propose P2TAG, a framework designed for few-shot node classification on TAGs with graph pre-training and prompting. P2TAG first pre-trains the LLM (LM) and graph neural network (GNN) on TAGs with self-supervised loss. To fully utilize the ability of LLMs, we adapt the masked LLMing objective for our framework. The pre-trained model is then used for the few-shot node classification with a mixed prompt method, which simultaneously considers both text and graph information. We conduct experiments on six real-world TAGs, including paper citation networks and product co-purchasing networks. Experimental results demonstrate that our proposed framework outperforms existing graph few-shot learning methods on these datasets with +18.98% ~ +35.98% improvements.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. The extreme classification repository: Multi-label datasets and code. http://manikvarma.org/downloads/XC/XMLRepository.html
  2. Cluster-gcn: An efficient algorithm for training deep and large graph convolutional networks. In KDD’19. 257–266.
  3. Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction. In ICLR’22.
  4. Electra: Pre-training text encoders as discriminators rather than generators. In ICLR’20.
  5. Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering. In NeurIPS’16. 3837–3845.
  6. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT’19 (1). 4171–4186.
  7. Graph prototypical networks for few-shot learning on attributed networks. In CIKM’20.
  8. GLM: General language model pretraining with autoregressive blank infilling. In ACL’22. 320–335.
  9. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML’17.
  10. Inductive Representation Learning on Large Graphs. In NeurIPS’17.
  11. Kaveh Hassani and Amir Hosein Khasahmadi. 2020. Contrastive multi-view representation learning on graphs. In ICML’20. 4116–4126.
  12. DeBERTa: Decoding-enhanced BERT with Disentangled Attention. In ICLR’21.
  13. Harnessing explanations: LLM-to-LM interpreter for enhanced text-attributed graph representation learning. arXiv preprint arXiv:2305 (2023).
  14. GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learner. In WWW’23. 737–746.
  15. GraphMAE: Self-Supervised Masked Graph Autoencoders. In KDD’22. 594–604.
  16. Kexin Huang and Marinka Zitnik. 2020. Graph meta learning via local subgraphs. NeurIPS’20 (2020).
  17. Prompt-based node feature extractor for few-shot learning on text-attributed graphs. arXiv preprint arXiv:2309.02848 (2023).
  18. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE transactions on neural networks and learning systems 33, 2 (2021), 494–514.
  19. Albert: A lite bert for self-supervised learning of language representations. In ICLR’20.
  20. Jure Leskovec and Christos Faloutsos. 2006. Sampling from large graphs. In KDD’06.
  21. Adsgnn: Behavior-graph augmented relevance modeling in sponsored search. In SIGIR’21. 223–232.
  22. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019).
  23. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017).
  24. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013).
  25. GCC: Graph contrastive coding for graph neural network pre-training. In KDD’20. 1150–1160.
  26. Sign: Scalable inception graph neural networks. arXiv preprint arXiv:2004.11198 7 (2020), 15.
  27. Prototypical networks for few-shot learning. In NeurIPS’17.
  28. Gppt: Graph pre-training and prompt tuning to generalize graph neural networks. In KDD’22. 1717–1727.
  29. All in One: Multi-Task prompting for graph neural networks. In KDD’23.
  30. Large-scale representation learning on graphs via bootstrapping. In ICLR’22.
  31. Graph Attention Networks. In ICLR’18.
  32. Microsoft academic graph: When experts are not enough. Quantitative Science Studies 1, 1 (2020), 396–413.
  33. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533 (2022).
  34. Task-adaptive few-shot node classification. In KDD’22.
  35. Max Welling and Thomas N Kipf. 2017. Semi-supervised classification with graph convolutional networks. In ICLR’17.
  36. Zhihao Wen and Yuan Fang. 2023. Augmenting low-Resource text classification with graph-grounded pre-training and prompting. In SIGIR’23.
  37. Simplifying graph convolutional networks. In ICML’19.
  38. Graph wavelet neural network. In ICLR’19.
  39. A comprehensive study on text-attributed graphs: Benchmarking and rethinking. In NeurIPS’23.
  40. Does Graph Distillation See Like Vision Dataset Counterpart?. In NeurIPS’23.
  41. GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph. In NeurIPS’21.
  42. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In NeurIPS’19. 5754–5764.
  43. Graph few-shot learning via knowledge transfer. In AAAI’20.
  44. LinkBERT: Pretraining Language Models with Document Links. In ACL’22.
  45. Graph convolutional neural networks for web-scale recommender systems. In KDD’18.
  46. Graph Contrastive Learning with Augmentations. In NeurIPS’20.
  47. Empower text-attributed graphs learning with large language models (llms). arXiv preprint arXiv:2310.09872 (2023).
  48. GraphSAINT: Graph Sampling Based Inductive Learning Method. In ICLR’20.
  49. Jiaqi Zeng and Pengtao Xie. 2021. Contrastive self-supervised learning for graph classification. In AAAI’21, Vol. 35. 10824–10832.
  50. From Canonical Correlation Analysis to Self-supervised Graph Neural Networks. In NeurIPS’21. 76–89.
  51. Learning on Large-scale Text-attributed Graphs via Variational Inference. In ICLR’23.
  52. Hierarchical label with imbalance and attributed network structure fusion for network embedding. AI Open 3 (2022), 91–100.
  53. Hierarchical representation learning for attributed networks. IEEE Transactions on Knowledge and Data Engineering 35, 3 (2023), 2641–2656.
  54. Rethinking and Scaling Up Graph Contrastive Learning: An Extremely Efficient Approach with Group Discrimination. In NeurIPS’22.
  55. Meta-gnn: On few-shot node classification in graph meta-learning. In CIKM’2019.
  56. Textgnn: Improving text encoder via graph neural network in sponsored search. In WWW’21. 2848–2857.
  57. Deep graph contrastive representation learning. arXiv preprint arXiv:2006.04131 (2020).
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now