ClinicalGPT: Large Language Models Finetuned with Diverse Medical Data and Comprehensive Evaluation (2306.09968v1)

Abstract: LLMs have exhibited exceptional performance on various NLP tasks, leveraging techniques such as the pre-training, and instruction fine-tuning. Despite these advances, their effectiveness in medical applications is limited, due to challenges such as factual inaccuracies, reasoning abilities, and lack grounding in real-world experience. In this study, we present ClinicalGPT, a LLM explicitly designed and optimized for clinical scenarios. By incorporating extensive and diverse real-world data, such as medical records, domain-specific knowledge, and multi-round dialogue consultations in the training process, ClinicalGPT is better prepared to handle multiple clinical task. Furthermore, we introduce a comprehensive evaluation framework that includes medical knowledge question-answering, medical exams, patient consultations, and diagnostic analysis of medical records. Our results demonstrate that ClinicalGPT significantly outperforms other models in these tasks, highlighting the effectiveness of our approach in adapting LLMs to the critical domain of healthcare.

• The paper presents ClinicalGPT's fine-tuning of BLOOM-7B using supervised and reinforcement learning to enhance clinical accuracy.
• It leverages diverse datasets such as cMedQA2, MedDialog, and MD-EHR to optimize medical dialogues and examination performance.
• It achieves high BLEU, ROUGE scores and an 80.9% diagnostic accuracy, underscoring robust performance in critical medical tasks.

ClinicalGPT: Fine-tuning LLMs for Medical Applications

LLMs have shown impressive capabilities across various NLP tasks. However, their application in the medical domain faces significant challenges due to the need for high accuracy, reasoning abilities, and real-world grounding. This essay discusses "ClinicalGPT: LLMs Finetuned with Diverse Medical Data and Comprehensive Evaluation" (2306.09968), which presents ClinicalGPT, an LLM specifically optimized for clinical tasks through extensive fine-tuning using diverse medical datasets.

Methodology

Dataset Composition

ClinicalGPT was trained using a wide range of datasets such as cMedQA2, cMedQA-KG, MD-EHR, MEDQA-MCMLE, and MedDialog. These datasets contributed various forms of medical data:

• cMedQA2: A Chinese medical question-answer dataset.
• cMedQA-KG: Developed from knowledge graphs focused on disease, medication, and symptom entities.
• MEDQA-MCMLE: Encompasses multiple-choice medical exam questions.
• MedDialog: Includes multi-turn dialogues resembling real doctor-patient interactions.
• MD-EHR: Electronic Health Records dataset covering various disease groups.

Fine-tuning Process

The training strategy entailed a robust instruction-tuning approach with Supervised Fine Tuning (SFT) and Reinforcement Learning (RL). Using the BLOOM-7B LLM as the base, ClinicalGPT was fine-tuned to generate accurate and coherent medical texts. This approach involved:

• Supervised Fine Tuning: Optimizing the likelihood generation of responses based on input prompts.
• Reinforcement Learning: Employing a reward model to enhance output quality, optimized with Proximal Policy Optimization (PPO) to manage divergence from initial states.

  Figure 1: The overview of ClinicalGPT.

Experimental Results

Medical Conversations

ClinicalGPT demonstrated superior performance in generating relevant and coherent responses in medical dialogues, as indicated by metrics such as BLEU and ROUGE. Specifically, it achieved the highest BLEU-1 and ROUGE scores, highlighting its proficiency in context preservation and informativeness.

Medical Examinations

When tested on the MEDQA-MCMLE dataset, ClinicalGPT outperformed other models across eight medical categories, particularly excelling in Rheumatic immune conditions. This indicates its strong clinical reasoning and knowledge integration capabilities for medical exams.

Diagnosis

In diagnosing conditions within the MD-EHR dataset, ClinicalGPT exceeded other models with an average accuracy of 80.9%. It displayed notable competence in the Digestive and Urinary categories, although there is room for improvement in Gynecology and Hematology.

Medical Question Answering

Using an evaluation benchmark with GPT-4, ClinicalGPT consistently surpassed BLOOM-7B, LLAMA-7B, and ChatGLM-6B in accuracy, helpfulness, and safety of responses. This confirms ClinicalGPT’s ability to generate reliable medical advice and information.

Conclusion

The development of ClinicalGPT highlights the potential of tailoring LLMs specifically for medical applications through comprehensive fine-tuning with clinically relevant data. The model demonstrates enhanced performance across various medical tasks, establishing a robust framework for deploying LLMs in clinical settings. Although ClinicalGPT shows promise in transforming medical AI applications, ongoing fine-tuning and adaptation are necessary to address specific domains and datasets, augmenting its capabilities as medical AI continues to evolve.