BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks (2305.17100v4)

Abstract: Traditional biomedical AI models, designed for specific tasks or modalities, often exhibit limited flexibility in real-world deployment and struggle to utilize holistic information. Generalist AI holds the potential to address these limitations due to its versatility in interpreting different data types and generating tailored outputs for diverse needs. However, existing biomedical generalist AI solutions are typically heavyweight and closed source to researchers, practitioners, and patients. Here, we propose BiomedGPT, the first open-source and lightweight vision-language foundation model, designed as a generalist capable of performing various biomedical tasks. BiomedGPT achieved state-of-the-art results in 16 out of 25 experiments while maintaining a computing-friendly model scale. We also conducted human evaluations to assess the capabilities of BiomedGPT in radiology visual question answering, report generation, and summarization. BiomedGPT exhibits robust prediction ability with a low error rate of 3.8% in question answering, satisfactory performance with an error rate of 8.3% in writing complex radiology reports, and competitive summarization ability with a nearly equivalent preference score to human experts. Our method demonstrates that effective training with diverse data can lead to more practical biomedical AI for improving diagnosis and workflow efficiency.

• The paper introduces a generalist model that integrates vision and language modalities to address a range of biomedical tasks with innovative multi-modal pretraining.
• It employs a transformer-based seq2seq architecture, combining VQ-GAN for images and BPE for text, and achieves superior results on 15 out of 25 medical benchmarks.
• The model demonstrates robust zero-shot transfer learning and competitive performance against larger models, signaling a promising step for AI in healthcare.

BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks

Introduction to BiomedGPT

BiomedGPT introduces a comprehensive vision-LLM designed for biomedical applications, leveraging the advancements in multi-modal AI to overcome the limitations of traditional, task-specific AI models in biomedicine. By training on diverse datasets and employing a unified architecture, BiomedGPT integrates vision and language capabilities to tackle a range of clinically significant tasks such as disease diagnosis and report generation.

Figure 1: The overview of BiomedGPT: workflow, performance, and pretraining datasets.

Architecture and Workflow

BiomedGPT is structured on a transformer-based sequence-to-sequence (seq2seq) model utilizing a BERT-style encoder and a GPT-style autoregressive decoder. This design facilitates the handling of multi-modal inputs through tokenization of diverse data types such as images and text using VQ-GAN for images and BPE for text. The model's architecture is augmented with task-specific instructions, ensuring adaptability across a wide array of biomedical tasks.

Training and Fine-tuning

Initial pretraining with a diverse set of tasks, including masked image modeling and multi-modal tasks like VQA and image captioning, allows BiomedGPT to develop versatile capabilities. The model is then fine-tuned on specific datasets covering five vital medical AI tasks, achieving state-of-the-art results on 15 out of 25 benchmarks, including superior performance on visual question answering and text summarization tasks.

Figure 2: BiomedGPT performs fine-tuning for vision-language downstream tasks.

Performance Analysis

Despite its relatively compact size, BiomedGPT achieves competitive accuracy across vision-language tasks, outperforming larger models like Med-PaLM M (12B) in key areas such as breast mass classification. The model demonstrates robust zero-shot transfer learning capabilities, suggesting potential as a scalable biomedical assistant.

Figure 3: BiomedGPT performs fine-tuning for uni-modal downstream tasks.

Ablation Study and Model Scaling

An ablation paper reveals the significance of maintaining diverse pretraining tasks to optimize downstream performance. The model's performance scales with size, highlighting potential gains from further enlarging the model under computational resources. However, challenges remain in balancing multi-task efficiency, particularly in the presence of domain-specific data imbalances.

Figure 4: Ablation paper to demonstrate the impact of diversity of pretraining datasets and tasks.

Zero-shot Learning and Evaluation

BiomedGPT's ability to perform zero-shot classification showcases its generalist design, with evaluations against models like GPT-4V. Human evaluations validate its readiness for medical applications, underscoring its potential to serve as a beneficial diagnostic and decision-making tool in real-world healthcare environments.

Figure 5: BiomedGPT generates the response via zero-shot transfer learning.

Conclusion

BiomedGPT represents a significant step toward generalist AI in biomedicine, integrating vision and language modalities into a unified framework that can efficiently tackle diverse medical tasks. The model's design and performance suggest a promising avenue for expanding AI's role in clinical settings, addressing both current limitations and potential expansions into new biomedical domains. Future work will focus on scaling the model and refining its capability to handle multi-modal inputs and complex biomedical queries, ensuring broader applicability and integration into healthcare systems.