Exploring Sparsity in Graph Transformers (2312.05479v1)
Abstract: Graph Transformers (GTs) have achieved impressive results on various graph-related tasks. However, the huge computational cost of GTs hinders their deployment and application, especially in resource-constrained environments. Therefore, in this paper, we explore the feasibility of sparsifying GTs, a significant yet under-explored topic. We first discuss the redundancy of GTs based on the characteristics of existing GT models, and then propose a comprehensive \textbf{G}raph \textbf{T}ransformer \textbf{SP}arsification (GTSP) framework that helps to reduce the computational complexity of GTs from four dimensions: the input graph data, attention heads, model layers, and model weights. Specifically, GTSP designs differentiable masks for each individual compressible component, enabling effective end-to-end pruning. We examine our GTSP through extensive experiments on prominent GTs, including GraphTrans, Graphormer, and GraphGPS. The experimental results substantiate that GTSP effectively cuts computational costs, accompanied by only marginal decreases in accuracy or, in some cases, even improvements. For instance, GTSP yields a reduction of 30\% in Floating Point Operations while contributing to a 1.8\% increase in Area Under the Curve accuracy on OGBG-HIV dataset. Furthermore, we provide several insights on the characteristics of attention heads and the behavior of attention mechanisms, all of which have immense potential to inspire future research endeavors in this domain.
- On Attention Redundancy: A Comprehensive Study. In ACL.
- NAGphormer: A Tokenized Graph Transformer for Node Classification in Large Graphs. In ICLR.
- Chasing sparsity in vision transformers: An end-to-end exploration. NeurIPS, 34: 19974–19988.
- A unified lottery ticket hypothesis for graph neural networks. In ICML.
- What Does BERT Look at? An Analysis of BERT’s Attention. In ACL Workshop.
- Analyzing Redundancy in Pretrained Transformer Models. In EMNLP.
- A Generalization of Transformer Networks to Graphs. AAAI Workshop.
- Rigging the lottery: Making all tickets winners. In ICML.
- Reducing Transformer Depth on Demand with Structured Dropout. In ICLR.
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. In ICLR.
- Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. JMLR, 22(1): 10882–11005.
- Open Graph Benchmark: Datasets for Machine Learning on Graphs. arXiv:2005.00687.
- Deep networks with stochastic depth. In ECCV.
- Rethinking Graph Lottery Tickets: Graph Sparsity Matters. In ICLR.
- Global Self-Attention as a Replacement for Graph Convolution. In SIGKDD.
- Learned Token Pruning for Transformers. In SIGKDD.
- Semi-Supervised Classification with Graph Convolutional Networks. In ICLR.
- Similarity of Neural Network Representations Revisited. In ICML.
- Rethinking Graph Transformers with Spectral Attention. In NeurIPS.
- EViT: Expediting Vision Transformers via Token Reorganizations. In ICLR.
- Comprehensive Graph Gradual Pruning for Sparse Training in Graph Neural Networks. IEEE TNNLS.
- Sparse training via boosting pruning plasticity with neuroregeneration. In NeurIPS.
- Survey on Graph Neural Network Acceleration: An Algorithmic Perspective. In IJCAI.
- Are sixteen heads really better than one? In NeurIPS.
- Transformer for Graphs: An Overview from Architecture Perspective. arXiv:2202.08455.
- Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications.
- Recipe for a General, Powerful, Scalable Graph Transformer. In NeurIPS.
- Self-Supervised Graph Transformer on Large-Scale Molecular Data. In NeurIPS.
- Measuring and testing dependence by correlation of distances. The Annals of Statistics.
- Attention is All you Need. In NeurIPS.
- Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. In ACL.
- Representing Long-Range Context for Graph Neural Networks with Global Attention. In NeurIPS.
- Do Transformers Really Perform Badly for Graph Representation? In NeurIPS.
- Early-Bird GCNs: Graph-Network Co-Optimization Towards More Efficient GCN Training and Inference via Drawing Early-Bird Lottery Tickets. In AAAI.
- Width & depth pruning for vision transformers. In AAAI.
- Hierarchical graph transformer with adaptive node sampling. In NeurIPS.
- Are More Layers Beneficial to Graph Transformers? In ICLR.
- To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv:1710.01878.
- Chuang Liu (71 papers)
- Yibing Zhan (73 papers)
- Xueqi Ma (13 papers)
- Liang Ding (159 papers)
- Dapeng Tao (28 papers)
- Jia Wu (93 papers)
- Wenbin Hu (50 papers)
- Bo Du (264 papers)