StructGPT: A General Framework for Large Language Model to Reason over Structured Data (2305.09645v2)

Abstract: In this paper, we study how to improve the zero-shot reasoning ability of LLMs~(LLMs) over structured data in a unified way. Inspired by the study on tool augmentation for LLMs, we develop an \emph{Iterative Reading-then-Reasoning~(IRR)} approach for solving question answering tasks based on structured data, called \textbf{StructGPT}. In our approach, we construct the specialized function to collect relevant evidence from structured data (\ie \emph{reading}), and let LLMs concentrate the reasoning task based on the collected information (\ie \emph{reasoning}). Specially, we propose an \emph{invoking-linearization-generation} procedure to support LLMs in reasoning on the structured data with the help of the external interfaces. By iterating this procedures with provided interfaces, our approach can gradually approach the target answer to a given query. Extensive experiments conducted on three types of structured data demonstrate the effectiveness of our approach, which can significantly boost the performance of ChatGPT and achieve comparable performance against the full-data supervised-tuning baselines. Our codes and data are publicly available at~\url{https://github.com/RUCAIBox/StructGPT}.

• The paper introduces an IRR mechanism that iteratively refines LLM outputs for structured data queries.
• It details a tool augmentation strategy that enables LLMs to effectively interact with knowledge graphs, tables, and databases.
• Empirical evaluations show improved Hits@1, accuracy, and execution performance in few-shot and zero-shot settings.

Overview of "StructGPT: A General Framework for LLM to Reason over Structured Data"

The paper "StructGPT: A General Framework for LLM to Reason over Structured Data" introduces StructGPT, a framework tailored to enhance the reasoning capabilities of LLMs when interacting with structured data. This framework leverages the concept of tool augmentation and establishes a novel Iterative Reading-then-Reasoning (IRR) mechanism. The primary goal is to improve question-answering tasks that hinge on structured data such as knowledge graphs, tables, and databases.

Key Components

StructGPT operates by defining specialized interfaces that enable LLMs to effectively read and reason over structured data. The IRR framework facilitates the gradual accumulation of relevant evidence through an invoking-linearization-generation process. This iterative approach allows LLMs to refine their answers progressively based on the gathered information. The framework demonstrates notable improvements in LLM performance under both few-shot and zero-shot settings.

Experimental Evaluation

Extensive experiments were conducted across multiple datasets covering three distinct types of structured data: knowledge graphs, tables, and databases. Empirical results showcase significant performance gains in reasoning tasks facilitated by StructGPT. The improvement is evident in metrics such as Hits@1, accuracy, and execution accuracy across tasks like KG-based QA (KGQA), Table-based QA (TableQA), and Text-to-SQL. StructGPT enhances LLMs' ability to reason over large-scale, diverse data structures, achieving performance that competes with fully supervised models.

Implications and Future Work

The proposed framework opens new avenues for scalable and efficient reasoning over structured data using LLMs. The implications are profound for applications requiring domain-specific knowledge and logical reasoning. Future developments could explore broader evaluation scenarios and adapt the methodology to other LLMs, potentially extending to data-to-text generation tasks.

By addressing the inherent limitations of LLMs in handling structured formats, StructGPT aligns with ongoing efforts to incorporate external knowledge resources, providing a versatile tool for enhancing AI capabilities in structured reasoning.