Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Residual-based Language Models are Free Boosters for Biomedical Imaging (2403.17343v3)

Published 26 Mar 2024 in cs.CV, cs.CL, and cs.LG

Abstract: In this study, we uncover the unexpected efficacy of residual-based LLMs as part of encoders for biomedical imaging tasks, a domain traditionally devoid of language or textual data. The approach diverges from established methodologies by utilizing a frozen transformer block, extracted from pre-trained LLMs, as an innovative encoder layer for the direct processing of visual tokens. This strategy represents a significant departure from the standard multi-modal vision-language frameworks, which typically hinge on language-driven prompts and inputs. We found that these LLMs could boost performance across a spectrum of biomedical imaging applications, including both 2D and 3D visual classification tasks, serving as plug-and-play boosters. More interestingly, as a byproduct, we found that the proposed framework achieved superior performance, setting new state-of-the-art results on extensive, standardized datasets in MedMNIST-2D and 3D. Through this work, we aim to open new avenues for employing LLMs in biomedical imaging and enriching the understanding of their potential in this specialized domain.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (82)
  1. A dataset of microscopic peripheral blood cell images for development of automatic recognition systems. Data in brief, 30, 2020.
  2. Dataset of breast ultrasound images. Data in brief, 28:104863, 2020.
  3. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716–23736, 2022.
  4. A completed reference database of lung nodules on ct scans. Academic Radiology, 14(12):1455–1463, 2007.
  5. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6836–6846, 2021.
  6. A survey of heterogeneous transfer learning, 2023.
  7. Making the most of text semantics to improve biomedical vision–language processing. In European conference on computer vision, pages 1–21. Springer, 2022.
  8. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  9. Automated data curation for robust language model fine-tuning. arXiv preprint arXiv:2403.12776, 2024.
  10. When do you need chain-of-thought prompting for chatgpt? arXiv preprint arXiv:2304.03262, 2023a.
  11. Alleviating data imbalance issue with perturbed input during inference. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part V 24, pages 407–417. Springer, 2021.
  12. Personalized fall risk assessment for long-term care services improvement. In 2017 Annual Reliability and Maintainability Symposium (RAMS), pages 1–7. IEEE, 2017a.
  13. Multi-state reliability demonstration tests. Quality Engineering, 29(3):431–445, 2017b.
  14. A data heterogeneity modeling and quantification approach for field pre-assessment of chloride-induced corrosion in aging infrastructures. Reliability Engineering & System Safety, 171:123–135, 2018.
  15. Claims data-driven modeling of hospital time-to-readmission risk with latent heterogeneity. Health care management science, 22:156–179, 2019.
  16. Optimal binomial reliability demonstration tests design under acceptance decision uncertainty. Quality Engineering, 32(3):492–508, 2020.
  17. Recontab: Regularized contrastive representation learning for tabular data. arXiv preprint arXiv:2310.18541, 2023b.
  18. Graph meets llm: A novel approach to collaborative filtering for robust conversational understanding. arXiv preprint arXiv:2305.14449, 2023c.
  19. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113, 2023.
  20. Transmed: Transformers advance multi-modal medical image classification. Diagnostics, 11(8):1384, 2021.
  21. DC Dataset. The 2nd diabetic retinopathy–grading and image quality estimation challenge, 2020.
  22. Weakly and semi-supervised deep level set network for automated skin lesion segmentation. In Innovation in Medicine and Healthcare: Proceedings of 8th KES-InMed 2020, pages 145–155. Springer, 2020.
  23. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
  24. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  25. Confidence trigger detection: an approach to build real-time tracking-by-detection system. arXiv preprint arXiv:1902.00615, 2019.
  26. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  27. Utnet: a hybrid transformer architecture for medical image segmentation. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part III 24, pages 61–71. Springer, 2021.
  28. From images to textual prompts: Zero-shot visual question answering with frozen large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10867–10877, 2023.
  29. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14953–14962, 2023.
  30. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  31. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
  32. How many validation labels do you need? exploring the design space of label-efficient model ranking, 2024.
  33. Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3942–3951, 2021.
  34. Analyzing entropy features in time-series data for pattern recognition in neurological conditions. Artificial Intelligence in Medicine, page 102821, 2024.
  35. Machine learning and predictive analytics: Advancing disease prevention in healthcare. Journal of Contemporary Healthcare Analytics, 7(1):53–71, 2023.
  36. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, pages 590–597, 2019.
  37. Perceiver: General perception with iterative attention. In International conference on machine learning, pages 4651–4664. PMLR, 2021.
  38. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
  39. Grounding language models to images for multimodal inputs and outputs. 2023.
  40. Adaptive ensembles of fine-tuned transformers for llm-generated text detection. arXiv preprint arXiv:2403.13335, 2024.
  41. A-tip: attribute-aware text infilling via pre-trained language model. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5857–5869, 2022a.
  42. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694–9705, 2021.
  43. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888–12900. PMLR, 2022b.
  44. Towards fast adaptation of pretrained contrastive models for multi-channel video-language retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14846–14855, 2023.
  45. T3d: Towards 3d medical image understanding through vision-language pre-training. arXiv preprint arXiv:2312.01529, 2023a.
  46. Parameter-efficient transfer learning for medical visual question answering. IEEE Transactions on Emerging Topics in Computational Intelligence, 2023b.
  47. A chatgpt aided explainable framework for zero-shot medical image diagnosis. arXiv preprint arXiv:2307.01981, 2023c.
  48. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021.
  49. Annotated high-throughput microscopy image sets for validation. Nature methods, 9(7):637–637, 2012.
  50. Linearly mapping from image to text space. arXiv preprint arXiv:2209.15162, 2022.
  51. Frozen transformers in language models are effective visual encoder layers. arXiv preprint arXiv:2310.12973, 2023.
  52. Elastic net nonparallel hyperplane support vector machine and its geometrical rationality. IEEE Transactions on Neural Networks and Learning Systems, 33(12):7199–7209, 2021.
  53. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  54. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.
  55. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36, 2024.
  56. Combining incremental conductance and firefly algorithm for tracking the global mpp of pv arrays. Journal of Renewable and Sustainable Energy, 9(2), 2017.
  57. Going through the motions: AR/VR keylogging from user head motions. In 32nd USENIX Security Symposium (USENIX Security 23), pages 159–174, Anaheim, CA, 2023. USENIX Association.
  58. Table meets llm: Can large language models understand structured table data? a benchmark and empirical study. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 645–654, 2024.
  59. Optimizing crop management with reinforcement learning and imitation learning. arXiv preprint arXiv:2209.09991, 2022.
  60. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  61. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1–9, 2018.
  62. Medical transformer: Gated axial-attention for medical image segmentation. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part I 24, pages 36–46. Springer, 2021.
  63. Optimal test design for reliability demonstration under multi-stage acceptance uncertainties. Quality Engineering, 0(0):1–14, 2023a.
  64. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36, 2024.
  65. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2097–2106, 2017.
  66. Emp: emotion-guided multi-modal fusion and contrastive learning for personality traits recognition. In Proceedings of the 2023 ACM International Conference on Multimedia Retrieval, pages 243–252, 2023b.
  67. Balanced training for sparse gans. In Thirty-seventh Conference on Neural Information Processing Systems, 2023c.
  68. Unleashing the power of graph learning through llm-based autonomous agents. arXiv preprint arXiv:2309.04565, 2023.
  69. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022.
  70. Optimizing nitrogen management with deep reinforcement learning and crop simulations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1712–1720, 2022.
  71. Hallucination improves the performance of unsupervised visual representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16132–16143, 2023a.
  72. Genco: An auxiliary generator from contrastive learning for enhanced few-shot learning in remote sensing. arXiv preprint arXiv:2307.14612, 2023b.
  73. Extended agriculture-vision: An extension of a large aerial image dataset for agricultural pattern analysis. arXiv preprint arXiv:2303.02460, 2023c.
  74. Switchtab: Switched autoencoders are effective tabular learners. arXiv preprint arXiv:2401.02013, 2024.
  75. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 10(1):41, 2023.
  76. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
  77. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF international conference on computer vision, pages 558–567, 2021.
  78. Medical image classification using synergic deep learning. Medical Image Analysis, 54:10–19, 2019.
  79. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022a.
  80. Patch-level contrastive learning via positional query for visual pre-training. In International Conference on Machine Learning, pages 41990–41999. PMLR, 2023.
  81. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference, pages 2–25. PMLR, 2022b.
  82. Visual in-context learning for large vision-language models. arXiv preprint arXiv:2402.11574, 2024.
Citations (24)
View on Semantic Scholar

Summary

  • The paper introduces the novel approach of incorporating frozen LLM blocks into visual transformer encoders to enhance biomedical image classification.
  • It utilizes additional trainable linear layers and residual connections for efficient dimension alignment and smooth information flow.
  • Empirical evaluations across multiple datasets, including BreastMNIST, RetinaMNIST, and DermaMNIST, establish new state-of-the-art performance benchmarks.

Unveiling the Potential of LLMs in Biomedical Imaging

The Novel Approach

In the field of biomedical imaging, the quest for models that can accurately interpret and classify images is ongoing. Traditional methodologies have leaned heavily on Vision Transformers (ViTs) and other AI technologies. However, challenges such as the need for vast, meticulously labeled datasets and the complexity of model optimization have remained significant hurdles. This paper introduces an innovative solution: leveraging the capabilities of pre-trained LLMs as a novel encoder layer within Visual Transformer architectures for biomedical imaging tasks. This approach diverges from convention by using LLMs not for text processing but for visual data interpretation, showcasing a new avenue for the efficacy of LLMs beyond their original domain.

Methodology

The core premise of this paper lies in the integration of a frozen transformer block from a pre-trained LLM into a vision-based encoder architecture. This is facilitated by additional trainable linear layers for dimension alignment and a residual connection to smooth the flow of information. Such an architecture subtly embeds the nuanced capabilities of LLMs into the visual data processing pipeline, enhancing the model's ability to grasp and interpret complex biomedical images.

Empirical Evaluation

The method's effectiveness is rigorously tested across several biomedical imaging tasks, both 2D and 3D. The researchers employed a variety of datasets, such as BreastMNIST, RetinaMNIST, DermaMNIST, and others, catering to different types of biomedical imaging challenges. The results are strikingly positive, with the LLM-equipped models consistently outperforming traditional ViT frameworks. Notably, the approach sets new state-of-the-art results on widely recognized benchmarks, demonstrating the potential of LLMs as robust enhancers of biomedical image analysis.

Insights and Contributions

This investigation not only validates the hypothesis that LLMs, even when detached from their initial linguistic confines, can significantly contribute to visual tasks but also elucidates several key findings:

  • Novelty in Application: The paper pioneers the use of frozen transformer blocks from LLMs as boosters in biomedical image encoders, laying groundwork for further exploration in this interdisciplinary niche.
  • Performance Gains: The approach notably surpasses existing benchmarks in biomedical image classification tasks, highlighted by strong numerical results across various datasets.
  • Flexibility and Efficiency: The method offers a plug-and-play solution that is adaptable to various data scales and types without the need for intensive computational resources or data.

Future Directions

The promising outcomes invite speculation on future developments in leveraging LLMs for specialized domains like biomedical imaging. There are several pathways for advancing this research:

  • Extending the application to broader datasets and learning tasks, possibly including tasks beyond image classification to encompass segmentation and anomaly detection.
  • Investigating the integration of LLM features that specifically exploit the unique qualities of biomedical images, such as the detailed textual descriptions found in medical reports.
  • Exploring the fine-tuning of frozen LLM blocks in a targeted manner to adapt more closely to the nuances of biomedical visual data.

Conclusion

The intersection of LLMs and visual data processing, as explored in this paper, marks a significant stride in the application of AI within the biomedical field. By turning to the untapped potential of LLMs for image analysis, this research not only challenges existing paradigms but also offers a beacon for future explorations aimed at enhancing the precision and efficiency of biomedical imaging tasks.