PathoDuet: Foundation Models for Pathological Slide Analysis of H&E and IHC Stains (2312.09894v2)

Abstract: Large amounts of digitized histopathological data display a promising future for developing pathological foundation models via self-supervised learning methods. Foundation models pretrained with these methods serve as a good basis for downstream tasks. However, the gap between natural and histopathological images hinders the direct application of existing methods. In this work, we present PathoDuet, a series of pretrained models on histopathological images, and a new self-supervised learning framework in histopathology. The framework is featured by a newly-introduced pretext token and later task raisers to explicitly utilize certain relations between images, like multiple magnifications and multiple stains. Based on this, two pretext tasks, cross-scale positioning and cross-stain transferring, are designed to pretrain the model on Hematoxylin and Eosin (H&E) images and transfer the model to immunohistochemistry (IHC) images, respectively. To validate the efficacy of our models, we evaluate the performance over a wide variety of downstream tasks, including patch-level colorectal cancer subtyping and whole slide image (WSI)-level classification in H&E field, together with expression level prediction of IHC marker, tumor identification and slide-level qualitative analysis in IHC field. The experimental results show the superiority of our models over most tasks and the efficacy of proposed pretext tasks. The codes and models are available at https://github.com/openmedlab/PathoDuet.

• The paper presents two novel pretext tasks—cross-scale positioning and cross-stain transferring—to improve slide analysis accuracy.
• It leverages Vision Transformer architectures with a pretext token mechanism to integrate local and global image context effectively.
• Experimental results show superior patch-level and whole slide classification, paving the way for scalable, annotation-efficient diagnostics.

An Overview of PathoDuet: Foundation Models for Pathological Slide Analysis

The paper "PathoDuet: Foundation Models for Pathological Slide Analysis of H&E and IHC Stains" introduces a framework and series of pretrained models called PathoDuet, which aim to enhance self-supervised learning (SSL) methodologies for analyzing histopathological slides. This essay provides an expert analysis of the paper's contributions, experimental results, and implications for future developments in computational pathology.

PathoDuet addresses the challenge of interpreting digitized histopathological data, particularly given the differences between natural and pathological images, which hinder the direct application of existing image processing methods. The framework introduces two novel pretext tasks—cross-scale positioning and cross-stain transferring—to bolster the model's ability to handle Hematoxylin and Eosin (H&E) stained images and transfer knowledge to immunohistochemistry (IHC) images.

Methodology

PathoDuet leverages Vision Transformer (ViT) architectures, introducing a pretext token mechanism to incorporate additional input forms required by the pretext tasks. This token allows the network to effectively process auxiliary data—such as variations in stain and magnification—without resorting to multiple networks.

1. Cross-Scale Positioning Task: This task emulates a pathologist's technique of zooming in and out of slides. It involves using a larger region to provide context for understanding a small patch, thereby balancing the focus on local and global information across different magnifications. This task is supported by a specialized positioning mechanism to weight features relative to their contextual importance.
2. Cross-Stain Transferring Task: This task facilitates transferring the pretrained H&E model's structural understanding to IHC images. By adopting adaptive instance normalization (AdaIN), the paper innovatively models the transfer of stylistic information (e.g., stain differences) between these modalities, thus aligning disparate imaging sources in a shared semantic space.

Experimental Results

The authors implemented extensive experimentation to validate PathoDuet. The framework was pretrained using extensive datasets from TCGA, HyReCo, and BCI, ensuring robustness across different tissue samples and stains.

• In the case of H&E image analysis, PathoDuet demonstrated superior performance over contemporary models by achieving higher accuracies in patch-level classification tasks (e.g., colorectal cancer subtyping) and whole slide image (WSI) classification. These results underline the framework's effectiveness in capturing both micro and macro histological features.
• For IHC images, PathoDuet efficiently adapted H&E knowledge to IHC staining, excelling in tasks such as tumor cell identification and the assessment of expression levels of markers like PD-L1. This adaptability is essential for broad applications in clinical settings where IHC insights complement initial H&E examinations.

Implications and Future Directions

The introduction of pretext tokens and task raisers in PathoDuet sets a precedent for the design of SSL frameworks tailored to specialized data types like pathological slides. This methodology can significantly lower the dependence on annotated data by maximizing the use of contextual and relational image features.

The theoretical implications suggest new pathways for neural architectures that integrate domain-specific knowledge directly into model pretraining phases. Practically, as digital pathology increasingly becomes a standard in diagnostics, frameworks like PathoDuet could streamline workflows and potentially improve diagnostic accuracy both in well-documented and resource-constrained settings.

Future research may focus on expanding these principles to other medical imaging modalities, such as MRI or CT, and exploring synergy effects with multimodal data inputs. Moreover, scaling up the foundation models with larger datasets and exploring integration with clinical data could yield models of unprecedented utility in personalized medicine.

In summary, PathoDuet represents a significant advancement in computational pathology, offering models adept at handling diverse staining techniques and magnification scales while reducing reliance on expert-labeled data. This work forms a basis for ongoing innovations in the analysis of medical images and the development of generalized AI models for healthcare diagnostics.