Increasing the LLM Accuracy for Question Answering: Ontologies to the Rescue! (2405.11706v1)

Abstract: There is increasing evidence that question-answering (QA) systems with LLMs, which employ a knowledge graph/semantic representation of an enterprise SQL database (i.e. Text-to-SPARQL), achieve higher accuracy compared to systems that answer questions directly on SQL databases (i.e. Text-to-SQL). Our previous benchmark research showed that by using a knowledge graph, the accuracy improved from 16% to 54%. The question remains: how can we further improve the accuracy and reduce the error rate? Building on the observations of our previous research where the inaccurate LLM-generated SPARQL queries followed incorrect paths, we present an approach that consists of 1) Ontology-based Query Check (OBQC): detects errors by leveraging the ontology of the knowledge graph to check if the LLM-generated SPARQL query matches the semantic of ontology and 2) LLM Repair: use the error explanations with an LLM to repair the SPARQL query. Using the chat with the data benchmark, our primary finding is that our approach increases the overall accuracy to 72% including an additional 8% of "I don't know" unknown results. Thus, the overall error rate is 20%. These results provide further evidence that investing knowledge graphs, namely the ontology, provides higher accuracy for LLM powered question answering systems.

• The paper introduces two key components, OBQC and LLM Repair, to enhance the accuracy of LLM-based QA systems through ontological validation.
• It leverages domain and range rules to correct SPARQL queries, achieving an overall accuracy of 72% and reducing error rates to 20%.
• The method is tested under various question and schema complexities, demonstrating robust performance improvements even in high-complexity scenarios.

Increasing the Accuracy of LLM Question-Answering Systems with Ontologies

The paper "Increasing the LLM Accuracy for Question Answering: Ontologies to the Rescue!" explores mechanisms to enhance the performance of LLM-based question-answering systems by incorporating ontologies into the query generation and correction process. This paper outlines two main components: Ontology-Based Query Check (OBQC) and LLM Repair, which work synergistically to reduce error rates and increase accuracy in complex querying scenarios.

Introduction

Question answering (QA) systems powered by LLMs have become increasingly popular in enterprise environments, especially for querying information from SQL databases. However, the raw Text-to-SQL approach often suffers from lower accuracy rates. Previous benchmark research revealed that utilizing a semantic representation of enterprise data through knowledge graphs significantly improved accuracy rates—from 16% to 54%. This research proposes additional improvements by utilizing the semantic capabilities of ontologies to detect and correct errors in LLM-generated SPARQL queries. The primary goal is to deliver more reliable answers by enforcing semantic constraints during query formulation and iteration.

Methodology

Ontology-Based Query Check (OBQC)

OBQC leverages the semantic definitions present in ontologies to verify the validity of SPARQL queries generated by LLMs. It comprises rule-based evaluations on the query components:

• Domain and Range Rules: These rules validate that the subjects and objects of triples used in queries comply with semantic definitions such as rdfs:domain and rdfs:range. Violations indicate mismatches in expected data types or structural logic.
• Double Domain and Range Rules: These rules check for conflicts when multiple properties apply conditions on shared subjects or objects. If two properties prescribe conflicting definitions, corrections are initiated based on established ontological hierarchies.
• Incorrect Property Checks: This verification ensures properties used in queries exist in the defined ontology, dismissing non-standard properties unless specifically defined.
• IRI vs. Human-readable Checks in SELECT Clauses: These checks ensure variables intended for display are not raw IRIs, which lack meaningful context for business users.

  Figure 1: Overview of our Ontology-based Query Checker and LLM Repair approach.

LLM Repair

Upon identification of incorrect queries by OBQC, explanations are generated on why the current SPARQL query doesn't align with the ontology. This explanation, alongside the incorrect query, is used to prompt the LLM to generate a corrected version. This iterative process continues until a valid query is formed or a predefined attempt limit is reached, beyond which an "unknown" state is acknowledged.

Experimental Setup

Using the context from previous works, the evaluation employs benchmarks focusing on variations in question and schema complexity: Low/High Question Complexity crossed with Low/High Schema Complexity. GPT-4 processes these with zero-shot prompting, generating SPARQL queries which the OBQC checks. If errors are found, the query is repaired iteratively as needed.

Results

The approach significantly improved the overall execution accuracy to 72%, with an error rate reduction to 20%:

Figure 2: Average Overall Execution Accuracy (AOEA) of SPARQL and SQL for all the questions in the benchmark.

• Low Question/Low Schema Complexity: The most pronounced improvement, reducing error rates to 10.46%.
• High Complexity Scenarios: Substantial accuracy increases reflecting the robustness of ontology-based checks in handling intricate schema dependencies.

  Figure 3: Average Overall Execution Accuracy (AOEA) of SPARQL and SQL for all questions in each quadrant.

The rule utility analysis revealed that most corrections involved domain rules, highlighting common LLM errors at triple beginnings versus more sporadic issues at ranges.

Conclusion

Integrating ontological checks into LLM QA systems delivers noticeable accuracy improvements. These systems can reliably decrease error rates by structurally enforcing semantic correctness in query generation. Future research may enhance the expressiveness of ontologies or extend rule applications to cater to logic-based OWL extensions. Overall, the ontology-augmented methods advance the goal of creating more trustworthy AI-driven data retrieval systems in organizational contexts, promising better adoption rates through enhanced reliability.