Graph-oriented Instruction Tuning of Large Language Models for Generic Graph Mining (2403.04780v3)

Abstract: Graphs with abundant attributes are essential in modeling interconnected entities and enhancing predictions across various real-world applications. Traditional Graph Neural Networks (GNNs) often require re-training for different graph tasks and datasets. Although the emergence of LLMs has introduced new paradigms in natural language processing, their potential for generic graph mining, training a single model to simultaneously handle diverse tasks and datasets, remains under-explored. To this end, our novel framework MuseGraph, seamlessly integrates the strengths of GNNs and LLMs into one foundation model for graph mining across tasks and datasets. This framework first features a compact graph description to encapsulate key graph information within language token limitations. Then, we propose a diverse instruction generation mechanism with Chain-of-Thought (CoT)-based instruction packages to distill the reasoning capabilities from advanced LLMs like GPT-4. Finally, we design a graph-aware instruction tuning strategy to facilitate mutual enhancement across multiple tasks and datasets while preventing catastrophic forgetting of LLMs' generative abilities. Our experimental results demonstrate significant improvements in five graph tasks and ten datasets, showcasing the potential of our MuseGraph in enhancing the accuracy of graph-oriented downstream tasks while improving the generation abilities of LLMs.

• The paper introduces MuseGraph, which employs graph-oriented instruction tuning to integrate LLMs with GNNs for effective graph mining.
• It leverages a compact graph description mechanism combined with Chain-of-Thought-based instruction packages to efficiently encode graph structure and semantics.
• MuseGraph demonstrates superior performance on node classification, link prediction, and graph-to-text generation across various datasets.

Graph-oriented Instruction Tuning of LLMs for Generic Graph Mining

The paper "Graph-oriented Instruction Tuning of LLMs for Generic Graph Mining" presents MuseGraph, a novel framework that effectively combines the capabilities of Graph Neural Networks (GNNs) and LLMs to tackle diverse graph mining tasks across various datasets. By leveraging instruction tuning specifically designed for graph-related data, MuseGraph introduces a unified approach to address the challenges of graph representation and reasoning in LLMs.

Introduction and Motivation

Graphs serve as fundamental structures for modeling interconnections among entities with rich attributes. Traditional GNNs excel at learning from these structures but face limitations when generalizing across different tasks and datasets without extensive retraining. The integration of LLMs, known for their proficiency in natural language tasks, holds potential for enhancing graph mining capabilities. However, the challenge lies in effectively bridging the gap between graph data and LLMs' proficiency in processing text, given the constraints of language token limits and the diversity of graph-related tasks (Figure 1).

Figure 1: An illustrative toy example of the need for a generic graph model that can be directly applied to various graph-related tasks and datasets.

MuseGraph Framework

Compact Graph Description

The compact graph description mechanism is central to MuseGraph, enabling efficient encoding of graph information within LLM-compatible language tokens. This component uses a combination of neighbor and walk-based strategies to effectively capture semantic and structural aspects of graphs. The process leverages a "node energy" metric that considers token counts and node degrees to prioritize graph information within token limits (Figure 2).

Figure 2: The overall of MuseGraph, which consists of Compact Graph Description, Diverse Instruction Generation, and Graph-aware Instruction Tuning.

Diverse Instruction Generation

To harness the reasoning capabilities of advanced LLMs such as GPT-4, MuseGraph implements a Chain-of-Thought (CoT)-based instruction package (Figure 3). This approach involves prompting GPT-4 with task-specific instructions to extract step-by-step reasoning, which is then distilled into instruction packages that enhance LLMs' understanding of graph data. This methodology contrasts with existing techniques by constructing CoT-based instructions directly, rather than merely using them for prompting.

Figure 3: A process showing how to conduct the Chain-of-Thought (CoT)-based instruction package for node classification. We leverage the reasoning ability distilled from advanced LLMs (e.g., GPT-4) and integrate them with task-specific instructions via a 1:10 mix ratio.

Graph-aware Instruction Tuning

The graph-aware instruction tuning mechanism prevents catastrophic forgetting while accommodating task and dataset variability. MuseGraph employs dynamic instruction allocation strategies, balancing CoT-based instructions across tasks and datasets according to their complexity, ensuring comprehensive model training (Figure 4).

Figure 4: Comprehensive performance of different models on various tasks and datasets.

Experimental Results

MuseGraph demonstrates superior performance across multiple graph mining tasks, outperforming state-of-the-art GNNs and LLMs in node classification, link prediction, and graph-to-text generation tasks. Its effectiveness is shown in tasks spanning various datasets like IMDB, Freebase, and more, highlighting its capacity for generalization and adaptability (Figure 5).

Figure 5: Accuracy Results for the Reachability and Max Sum Path tasks on the graph with varying difficulty levels (i.e., D1 to D4), where ``D'' represents different degrees of complexity.

Conclusions and Future Work

MuseGraph offers a robust solution for generic graph mining by seamlessly unifying GNNs' representational strengths with LLMs' generative prowess. It opens avenues for deploying a single model capable of handling diverse graph-related tasks without extensive retraining. Future extensions can explore the integration of MuseGraph into a broader range of graph types and tasks, potentially extending its application to biological networks and knowledge graphs, enhancing versatility in real-world scenarios.