RadioRAG: Factual Large Language Models for Enhanced Diagnostics in Radiology Using Dynamic Retrieval Augmented Generation

RadioRAG: Factual LLMs for Enhanced Diagnostics in Radiology Using Dynamic Retrieval Augmented Generation

Introduction

The paper "RadioRAG: Factual LLMs for Enhanced Diagnostics in Radiology Using Dynamic Retrieval Augmented Generation" investigates a novel implementation of Retrieval Augmented Generation (RAG) tailored for radiology-specific inquiries. The approach is designed to address persistent issues related to the factual accuracy and up-to-dateness of information generated by LLMs in the medical domain.

Motivation and Background

LLMs like GPT-4 and Llama3 have demonstrated potential in various facets of clinical workflows, from automated machine learning for clinical data interpretation to structured data extraction from free-text reports. Despite these advancements, one of the main persistent challenges is their reliance on static and potentially outdated training data, which can result in the generation of inaccurate or biased information. Conventional strategies such as human feedback mechanisms and prompt engineering do not fully mitigate these challenges. This necessitates an innovative approach to foster dynamic interaction with real-time data sources, leading to the conception of Retrieval Augmented Generation (RAG).

RadioRAG Framework

RadioRAG represents an end-to-end framework that leverages RAG to enhance diagnostic accuracy in radiology. Unlike preceding RAG systems that rely on pre-compiled static databases, RadioRAG dynamically retrieves and integrates information from authoritative radiological sources such as www.radiopaedia.org in real-time. The framework is assessed using two novel datasets: RSNA-RadioQA, derived from the Radiological Society of North America (RSNA) Case Collection, and RadioQA, an expert-curated dataset designed to minimize data contamination from training sets.

Methodology

The framework consists of multiple components:

1. Key-phrase Extraction: The system employs GPT-3.5-turbo to extract up to five key-phrases from user queries, enhancing the specificity and relevance of the subsequent retrieval process.
2. Online Context Retrieval: Using these key-phrases, the system searches relevant articles from radiopaedia.org, which are transformed into vector embeddings and stored in a dynamically created vector database.
3. Contextual Retriever: The user query is converted into a vector and compared with the stored vectors to retrieve the top three most similar contexts.
4. LLM Response Generation: The LLM is then prompted to provide answers leveraging the retrieved context, which increases the factuality and relevance of the response.

Evaluation

RadioRAG's efficacy was evaluated using a comprehensive dataset that spans multiple radiological subspecialties, including breast imaging, musculoskeletal, neuroradiology, and oncologic imaging.

Model Performance:

• RadioRAG enhanced the diagnostic accuracy across all tested LLMs.
• GPT-4 and GPT-3.5-turbo saw increases in diagnostic accuracy with improvements ranging from 2% to 11%.
• Open-source models like Mixtral-8x7B-instruct-v0.1 and Llama3-8B demonstrated significant accuracy gains up to 47% and 33%, respectively, making them competitive with more complex models like GPT-4 in radiological contexts.

Statistical Analysis:

• The use of bootstrapping with 10,000 redraws and adjusted p-values confirmed the statistical significance of the results.
• RadioRAG's improvement in diagnostic accuracy, especially among open-source models, underlines its potential for cost-effective application in medical diagnostics without necessitating extensive retraining.

Implications and Future Work

The implications of RadioRAG are substantial. From a practical perspective, the framework offers a scalable solution for integrating real-time, authoritative data into LLMs to enhance the factual accuracy of medical diagnostics. Theoretically, RadioRAG provides insights into how LLMs can serve as dynamic reasoning engines rather than static repositories of pre-encoded knowledge. Future research directions include refining embedding functions and enhancing retrieval methodologies to further minimize inaccuracies. Additionally, optimization strategies to streamline real-time context retrieval processes and mitigate potential website load issues will be critical for clinical implementation.

Conclusion

RadioRAG sets a new benchmark for LLM applications in radiology by leveraging dynamic RAG to bridge the gap between static training data and real-time, factually accurate medical information. This framework not only enhances the diagnostic capabilities of LLMs but also paves the way for future developments in AI-driven diagnostics, significantly impacting clinical practices and patient care. The publicly available datasets—RSNA-RadioQA and RadioQA—further contribute to the transparency and reproducibility of research in this domain.