Don't Forget to Connect! Improving RAG with Graph-based Reranking (2405.18414v1)

Abstract: Retrieval Augmented Generation (RAG) has greatly improved the performance of LLM responses by grounding generation with context from existing documents. These systems work well when documents are clearly relevant to a question context. But what about when a document has partial information, or less obvious connections to the context? And how should we reason about connections between documents? In this work, we seek to answer these two core questions about RAG generation. We introduce G-RAG, a reranker based on graph neural networks (GNNs) between the retriever and reader in RAG. Our method combines both connections between documents and semantic information (via Abstract Meaning Representation graphs) to provide a context-informed ranker for RAG. G-RAG outperforms state-of-the-art approaches while having smaller computational footprint. Additionally, we assess the performance of PaLM 2 as a reranker and find it to significantly underperform G-RAG. This result emphasizes the importance of reranking for RAG even when using LLMs.

• The paper introduces G-RAG, which employs Graph Neural Networks and AMR graphs to model document interrelations and enhance reranking in RAG systems.
• It demonstrates improved accuracy with up to a 7 percentage point increase on datasets like Natural Questions and TriviaQA using a pairwise ranking loss.
• The results emphasize practical benefits for open-domain question answering and pave the way for future work on advanced GNN architectures and fine-tuning of LLM rerankers.

An Analysis of Retrieval Augmented Generation with GNN Rerankers in Open-Domain Question Answering

This paper introduces G-RAG, an advanced reranking mechanism aimed at enhancing Retrieval Augmented Generation (RAG) systems for Open-Domain Question Answering (ODQA). RAG enhances LLM outputs by integrating context from retrieved documents. However, traditional RAG struggles when dealing with documents that partially relate to the query or possess implicit connections requiring a sophisticated understanding of document interrelationships. G-RAG addresses these shortcomings by leveraging Graph Neural Networks (GNNs) to model and exploit connections across documents, incorporating Abstract Meaning Representation (AMR) graphs for semantic depth.

Methodology: Graph-Based Reranking

The core of this paper lies in its novel use of document graphs and AMR graphs to inform reranking processes within RAG frameworks. The approach involves several key steps:

1. Document Graph Construction: Document nodes are connected in an undirected graph based on shared concepts parsed through AMR. Edges between documents represent shared semantic content, ensuring that the graph reflects deeper inter-document relationships.
2. AMR Parsing and Node Features: AMR graphs are generated using AMRBART, capturing the semantic connections between query-document pairs. Document embeddings are supplemented by AMR-derived paths to encode richer context.
3. GNN Architecture: Node and edge features in the document graph are updated using GNNs. This setup employs the Graph Convolutional Network (GCN) architecture with Mean Aggregator and specific parameter tuning for optimal performance.
4. Ranking Mechanism: The model applies pairwise ranking loss to recalibrate document ranks, subsequently improving retrieval accuracy. This approach expressly addresses the challenge of identifying partially relevant documents that other methods might overlook.

Evaluation Metrics

The paper introduces Mean Tied Reciprocal Ranking (MTRR) and Tied Mean Hits@10 (TMHits@10) as innovative metrics to better evaluate reranking accuracy, particularly under scenarios involving tied relevance scores. These metrics ensure a more realistic assessment of reranker performance.

Experimental Results

Experiments conducted on the Natural Questions (NQ) and TriviaQA (TQA) datasets demonstrate G-RAG's capabilities. Key findings include:

• Performance: G-RAG-RL, which incorporates pairwise ranking loss, achieves superior performance compared to state-of-the-art methods like BART-GST and various baseline LLMs, sometimes showing improvements up to 7 percentage points in evaluation metrics.
• Embedding Models: Utilizing recent embedding models such as Ember enhanced reranking results significantly. Hyperparameter tuning yielded even better results, highlighting the importance of optimizing model parameters.
• LLMs as Rerankers: Experiments reveal that LLMs like PaLM~2 underperform in reranking tasks when applied naively without fine-tuning. The frequent occurrence of tied relevance scores with LLMs further underscores the necessity for specialized reranking strategies.

Theoretical and Practical Implications

The theoretical novelty of this research lies in merging graph-based document interrelations with reranking methods in ODQA. Practically, G-RAG has potential applications across various systems requiring precise information retrieval and context-aware generation.

Future Directions

Speculation on future advancements includes:

• Refinement of GNN Architectures: Exploring more sophisticated GNN variants or hybrid models could further enhance reranking capabilities.
• Advanced AMR Utilization: More efficient integration of AMR information could optimize computational footprints while retaining semantic accuracy.
• LLM Fine-Tuning: Adequate fine-tuning protocols for LLMs in reranking contexts could harness their generative prowess more effectively.

In conclusion, G-RAG introduces a robust, GNN-based reranking method that significantly advances the effectiveness of RAG systems in ODQA. By combining document interrelations with nuanced semantic representations, this research takes a pivotal step toward more accurate and context-aware LLM outputs. Future advancements in this domain are likely to build upon and refine these innovative strategies, promising further improvements in information retrieval and generation systems.