Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

HVTSurv: Hierarchical Vision Transformer for Patient-Level Survival Prediction from Whole Slide Image (2306.17373v1)

Published 30 Jun 2023 in cs.CV and cs.AI

Abstract: Survival prediction based on whole slide images (WSIs) is a challenging task for patient-level multiple instance learning (MIL). Due to the vast amount of data for a patient (one or multiple gigapixels WSIs) and the irregularly shaped property of WSI, it is difficult to fully explore spatial, contextual, and hierarchical interaction in the patient-level bag. Many studies adopt random sampling pre-processing strategy and WSI-level aggregation models, which inevitably lose critical prognostic information in the patient-level bag. In this work, we propose a hierarchical vision Transformer framework named HVTSurv, which can encode the local-level relative spatial information, strengthen WSI-level context-aware communication, and establish patient-level hierarchical interaction. Firstly, we design a feature pre-processing strategy, including feature rearrangement and random window masking. Then, we devise three layers to progressively obtain patient-level representation, including a local-level interaction layer adopting Manhattan distance, a WSI-level interaction layer employing spatial shuffle, and a patient-level interaction layer using attention pooling. Moreover, the design of hierarchical network helps the model become more computationally efficient. Finally, we validate HVTSurv with 3,104 patients and 3,752 WSIs across 6 cancer types from The Cancer Genome Atlas (TCGA). The average C-Index is 2.50-11.30% higher than all the prior weakly supervised methods over 6 TCGA datasets. Ablation study and attention visualization further verify the superiority of the proposed HVTSurv. Implementation is available at: https://github.com/szc19990412/HVTSurv.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. Divide-and-rule: self-supervised learning for survival analysis in colorectal cancer. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 480–489. Springer.
  2. Multiple Instance Learning with Mixed Supervision in Gleason Grading. arXiv preprint arXiv:2206.12798.
  3. The logrank test. Bmj, 328(7447): 1073.
  4. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature medicine, 1301–1309.
  5. Incorporating intratumoral heterogeneity into weakly-supervised deep learning models via variance pooling. arXiv preprint arXiv:2206.08885.
  6. Whole Slide Images are 2D Point Clouds: Context-Aware Survival Prediction using Patch-based Graph Convolutional Networks. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 339–349. Springer.
  7. Multimodal Co-Attention Transformer for Survival Prediction in Gigapixel Whole Slide Images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 4015–4025.
  8. Pan-cancer integrative histology-genomic analysis via multimodal deep learning. Cancer Cell.
  9. Multiple Instance Learning with Center Embeddings for Histopathology Classification. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 519–528.
  10. Big-Hypergraph Factorization Neural Network for Survival Prediction from Whole Slide Image. IEEE Transactions on Image Processing.
  11. Learning Visual Features by Colorization for Slide-Consistent Survival Prediction from Whole Slide Images. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 592–601. Springer.
  12. Multi-scale domain-adversarial multiple-instance CNN for cancer subtype classification with unannotated histopathological images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3852–3861.
  13. Survival model predictive accuracy and ROC curves. Biometrics, 61(1): 92–105.
  14. H2-MIL: Exploring Hierarchical Representation with Heterogeneous Multiple Instance Learning for Whole Slide Image Analysis.
  15. Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer. arXiv preprint arXiv:2106.03650.
  16. Integration of patch features through self-supervised learning and transformer for survival analysis on whole slide images. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 561–570. Springer.
  17. Weakly-supervised learning for lung carcinoma classification using deep learning. Scientific reports, 1–11.
  18. Nonparametric estimation from incomplete observations. Journal of the American statistical association, 53(282): 457–481.
  19. Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study. PLoS medicine, 16(1): e1002730.
  20. Challenge for diagnostic assessment of deep learning algorithm for metastases classification in sentinel lymph nodes on frozen tissue section digital slides in women with breast cancer. Cancer Research and Treatment: Official Journal of Korean Cancer Association, 52(4): 1103–1111.
  21. Weakly supervised multiple instance learning histopathological tumor segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 470–479.
  22. Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
  23. Graph CNN for survival analysis on whole slide pathological images. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 174–182. Springer.
  24. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 10012–10022.
  25. Manual tumor annotations in TCGA.
  26. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature Biomedical Engineering, 5(6): 555–570.
  27. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence, 42(4): 824–836.
  28. EPIC-Survival: End-to-end Part Inferred Clustering for Survival Analysis, with Prognostic Stratification Boosting. In Medical Imaging with Deep Learning, 520–531. PMLR.
  29. Deep learning-enabled breast cancer hormonal receptor status determination from base-level H&E stains. Nature communications, 1–8.
  30. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12: 2825–2830.
  31. A novel digital score for abundance of tumour infiltrating lymphocytes predicts disease free survival in oral squamous cell carcinoma. Scientific reports, 9(1): 1–13.
  32. Weakly supervised deep ordinal cox model for survival prediction from whole-slide pathological images. IEEE Transactions on Medical Imaging, 40(12): 3739–3747.
  33. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in Neural Information Processing Systems, 34.
  34. Explainable Survival Analysis with Convolution-Involved Vision Transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 2207–2215.
  35. Deep neural network models for computational histopathology: A survey. Medical Image Analysis, 67: 101813.
  36. Attention-Based Deep Neural Networks for Detection of Cancerous and Precancerous Esophagus Tissue on Histopathological Slides. JAMA Network Open.
  37. Long-term cancer survival prediction using multimodal deep learning. Scientific Reports, 11(1): 1–12.
  38. Attention is all you need. Advances in neural information processing systems, 30.
  39. Intratumoral heterogeneity in cancer progression and response to immunotherapy. Nature medicine, 27(2): 212–224.
  40. Hierarchical Graph Pathomic Network for Progression Free Survival Prediction. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 227–237. Springer.
  41. Wright, L. 2019. Ranger - a synergistic optimizer. https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer.
  42. Rethinking and improving relative position encoding for vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 10033–10041.
  43. CAMEL: A Weakly Supervised Learning Framework for Histopathology Image Segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 10681–10690.
  44. Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Medical Image Analysis, 65: 101789.
  45. Bias in cross-entropy-based training of deep survival networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(9): 3126–3137.
  46. Wsisa: Making survival prediction from whole slide histopathological images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 7234–7242.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Zhuchen Shao (6 papers)
  2. Yang Chen (535 papers)
  3. Hao Bian (8 papers)
  4. Jian Zhang (543 papers)
  5. Guojun Liu (7 papers)
  6. Yongbing Zhang (58 papers)
Citations (12)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now