Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Deep Weakly-Supervised Learning Methods for Classification and Localization in Histology Images: A Survey (1909.03354v7)

Published 8 Sep 2019 in cs.CV, cs.LG, and eess.IV

Abstract: Using deep learning models to diagnose cancer from histology data presents several challenges. Cancer grading and localization of regions of interest (ROIs) in these images normally relies on both image- and pixel-level labels, the latter requiring a costly annotation process. Deep weakly-supervised object localization (WSOL) methods provide different strategies for low-cost training of deep learning models. Using only image-class annotations, these methods can be trained to classify an image, and yield class activation maps (CAMs) for ROI localization. This paper provides a review of state-of-art DL methods for WSOL. We propose a taxonomy where these methods are divided into bottom-up and top-down methods according to the information flow in models. Although the latter have seen limited progress, recent bottom-up methods are currently driving much progress with deep WSOL methods. Early works focused on designing different spatial pooling functions. However, these methods reached limited localization accuracy, and unveiled a major limitation -- the under-activation of CAMs which leads to high false negative localization. Subsequent works aimed to alleviate this issue and recover complete object. Representative methods from our taxonomy are evaluated and compared in terms of classification and localization accuracy on two challenging histology datasets. Overall, the results indicate poor localization performance, particularly for generic methods that were initially designed to process natural images. Methods designed to address the challenges of histology data yielded good results. However, all methods suffer from high false positive/negative localization. Four key challenges are identified for the application of deep WSOL methods in histology -- under/over activation of CAMs, sensitivity to thresholding, and model selection.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (176)
  1. Sanity checks for saliency maps. In NeurIPS, 2018.
  2. innvestigate neural networks! JMLR, 20:93:1–93:8, 2019.
  3. Bach: Grand challenge on breast cancer histology images. CoRR, abs/1808.04277, 2018.
  4. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.
  5. P. Bamford and B. Lovell. Method for accurate unsupervised cell nucleus segmentation. In Intern. Conf. of the IEEE Engineering in Medicine and Biology Society, 2001.
  6. Bayesian belief networks in quantitative histopathology. Analytical and Quantitative Cytology and Histology, 14(6):459–473, 1992.
  7. Constrained domain adaptation for segmentation. In MICCAI, 2019.
  8. Network dissection: Quantifying interpretability of deep visual representations. In CVPR, 2017.
  9. What’s the point: Semantic segmentation with point supervision. In ECCV, 2016.
  10. D.M. Beck and S. Kastner. Top-down and bottom-up mechanisms in biasing competition in the human brain. Vision research, 49(10):1154–1165, Jun 2009.
  11. Min-max entropy for weakly supervised pointwise localization. CoRR, abs/1907.12934, 2019.
  12. Deep active learning for joint classification & segmentation with weak annotator. In WACV, 2021.
  13. Negative evidence matters in interpretable histology image classification. In Medical Imaging with Deep Learning (MIDL), 2022a.
  14. Deep interpretable classification and weakly-supervised segmentation of histology images via max-min uncertainty. IEEE Transactions on Medical Imaging, 41:702–714, 2022b.
  15. F-cam: Full resolution class activation maps via guided parametric upscaling. In WACV, 2022c.
  16. Ftcam: Temporal class activation maps for object localization in weakly-labeled unconstrained videos. In WACV, 2023.
  17. Cell-graph mining for breast tissue modeling and classification. In Intern. Conf. of the IEEE Engineering in Medicine and Biology Society, 2007.
  18. Content-based histopathology image retrieval using a kernel-based semantic annotation framework. Journal of Biomedical Informatics, 44(4):519–528, 2011.
  19. Quantification of tumour budding, lymphatic vessel density and invasion through image analysis in colorectal cancer. Journal of Translational Medicine, 12(1):156, 2014.
  20. Look and think twice: Capturing top-down visual attention with feedback convolutional neural networks. In ICCV, 2015.
  21. Multiple instance learning: A survey of problem characteristics and applications. Pattern Recognition, 77:329–353, 2018.
  22. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In WACV, 2018.
  23. Dcan: deep contour-aware networks for accurate gland segmentation. In CVPR, 2016.
  24. Semantic image segmentation with deep convolutional nets and fully connected crfs. In ICLR, 2015.
  25. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. PAMI, 40(4):834–848, 2018.
  26. Not-so-supervised: A survey of semi-supervised, multi-instance, and transfer learning in medical image analysis. Medical Image Analysis, 54:280–296, 2019.
  27. J. Choe and H. Shim. Attention-based dropout layer for weakly supervised object localization. In CVPR, 2019.
  28. Evaluating weakly supervised object localization methods right. In CVPR, 2020.
  29. Mitosis detection in breast cancer histology images with deep neural networks. In MICCAI, 2013.
  30. Classification and disease localization in histopathology using only global labels: A weakly-supervised approach. CoRR, abs/1802.02212, 2018.
  31. A deep learning architecture for image representation, visual interpretability and automated basal-cell carcinoma cancer detection. In MICCAI, 2013.
  32. P. Dabkowski and Y. Gal. Real time image saliency for black box classifiers. In NeurIPS, 2017.
  33. Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation. In ICCV, 2015.
  34. K. Daisuke and I. Shumpei. Machine learning methods for histopathological image analysis. Computational and Structural Biotechnology Journal, 16, 2018.
  35. A deep learning interpretable classifier for diabetic retinopathy disease grading. Neurocomputing, 396:465–476, 2020.
  36. S. Desai and H.G. Ramaswamy. Ablation-cam: Visual explanations for deep convolutional network via gradient-free localization. In WACV, 2020.
  37. R. Desimone. Visual attention mediated by biased competition in extrastriate visual cortex. Philosophical transactions of the Royal Society of London. Series B, Biological sciences, 353(1373):1245–1255, Aug 1998.
  38. R. Desimone and J. Duncan. Neural mechanisms of selective visual attention. Annual Review of Neuroscience, 18(1):193–222, 1995.
  39. T. Devries and G. W. Taylor. Improved regularization of convolutional neural networks with cutout. CoRR, abs/1708.04552, 2017.
  40. Deep learning for whole slide image analysis: An overview. Frontiers in Medicine, 6:264, 2019.
  41. 3D fully convolutional networks for subcortical segmentation in MRI: A large-scale study. NeuroImage, 2018.
  42. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
  43. Detecting prostatic adenocarcinoma from digitized histology using a multi-scale hierarchical classification approach. In Intern. Conf. of the IEEE Engineering in Medicine and Biology Society, 2006.
  44. Weldon: Weakly supervised learning of deep convolutional neural networks. In CVPR, 2016.
  45. Wildcat: Weakly supervised learning of deep convnets for image classification, pointwise localization and segmentation. In CVPR, 2017.
  46. Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer. Journal of the American Medical Association, 318(22):2199–2210, 2017.
  47. Microscopic fine-grained instance classification through deep attention. In MICCAI, Lecture Notes in Computer Science, 2020.
  48. Discriminative localization in cnns for weakly-supervised segmentation of pulmonary nodules. In MICCAI, 2017.
  49. Understanding deep networks via extremal perturbations and smooth masks. In ICCV, 2019.
  50. Interpretable explanations of black boxes by meaningful perturbation. In ICCV, 2017.
  51. B. Frenay and M. Verleysen. Classification in the presence of label noise: A survey. TNNLS, 25, 2014.
  52. Axiom-based grad-cam: Towards accurate visualization and explanation of cnns. In BMVC, 2020.
  53. TS-CAM: token semantic coupled attention map for weakly supervised object localization. In ICCV, 2021.
  54. Machine learning approaches to analyze histological images of tissues from radical prostatectomies. Computerized Medical Imaging and Graphics, 46, 2015.
  55. S. Ghosal and P. Shah. Interpretable and synergistic deep learning for visual explanation and statistical estimations of segmentation of disease features from medical images. CoRR, abs/2011.05791, 2020.
  56. Understanding integrated gradients with smoothtaylor for deep neural network attribution. CoRR, abs/2004.10484, 2020.
  57. Weakly-supervised localization of diabetic retinopathy lesions in retinal fundus images. In Int. Conf. on Image Processing, 2017.
  58. Deep Learning. MIT Press, 2016. http://www.deeplearningbook.org.
  59. Image analysis for neuroblastoma classification: segmentation of cell nuclei. In Intern. Conf. of the IEEE Engineering in Medicine and Biology Society, 2006.
  60. Histopathological image analysis: A review. IEEE reviews in Biomedical Engineering, 2:147, 2009.
  61. Resolving challenges in deep learning-based analyses of histopathological images using explanation methods. Scientific Reports, 10(1):6423, 2020.
  62. Expert system support using bayesian belief networks in the diagnosis of fine needle aspiration biopsy specimens of the breast. Journal of clinical pathology, 47, 1994.
  63. Page-net: Interpretable and integrative deep learning for survival analysis using histopathological images and genomic data. In Pacific Symposium on Biocomputing, 2019.
  64. Deep residual learning for image recognition. In CVPR, 2016.
  65. Histology image analysis for carcinoma detection and grading. Computer Methods and Programs in Biomedicine, 107(3):538–556, 2012.
  66. Why a pathology image should not be considered as a radiology image. Journal of Pathology Informatics, 2, 2011.
  67. Patch-based convolutional neural network for whole slide tissue image classification. In CVPR, 2016.
  68. Deep learning models for histopathological classification of gastric and colonic epithelial tumours. Scientific Reports, 10(1):1–11, 2020.
  69. Attention-based deep multiple instance learning. In ICML, 2018.
  70. Weakly-supervised learning-based feature localization in confocal laser endomicroscopy glioma images. CoRR, abs/1804.09428, 2018.
  71. Spatial transformer networks. In NIPS, 2015.
  72. A. Janowczyk and A. Madabhushi. Deep learning for digital pathology image analysis: A comprehensive tutorial with selected use cases. Journal of Pathology Informatics, 7, 2016.
  73. Constrained deep weak supervision for histopathology image segmentation. IEEE Transactions on Medical Imaging, 36(11):2376–2388, 2017.
  74. Layercam: Exploring hierarchical class activation maps for localization. IEEE Trans. Image Process., 30:5875–5888, 2021.
  75. M. Kandemir and F.A. Hamprecht. Computer-aided diagnosis from weak supervision: A benchmarking study. Computerized Medical Imaging and Graphics, 42:44–50, 2015.
  76. Curriculum semi-supervised segmentation. In MICCAI, 2019a.
  77. Constrained-cnn losses for weakly supervised segmentation. Medical Image Analysis, 54:88–99, 2019b.
  78. Simple does it: Weakly supervised instance and semantic segmentation. In CVPR, 2017.
  79. In-sample contrastive learning and consistent attention for weakly supervised object localization. In ACCV, 2020.
  80. Convolutional neural networks for histopathology image classification: Training vs. using pre-trained networks. In Int. Conf. on Image Processing Theory, Tools and Applications, 2017.
  81. Two-phase learning for weakly supervised object localization. In ICCV, 2017.
  82. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In ICML, 2020.
  83. The (Un)reliability of Saliency Methods, pages 267–280. Springer International Publishing, 2019.
  84. Looking under the hood: Deep neural network visualization to interpret whole-slide image analysis outcomes for colorectal polyps. In CVPR workshops, 2017.
  85. Imagenet classification with deep convolutional neural networks. In NeurIPS. 2012.
  86. Ficklenet: Weakly and semi-supervised semantic image segmentation using stochastic inference. In CVPR, 2019.
  87. Tell me where to look: Guided attention inference network. In CVPR, 2018.
  88. Y. Li and W. Ping. Cancer metastasis detection with neural conditional random field. In Medical Imaging with Deep Learning, 2018.
  89. Scribblesup: Scribble-supervised convolutional networks for semantic segmentation. In CVPR, 2016.
  90. Network in network. coRR, abs/1312.4400, 2013.
  91. A survey on deep learning in medical image analysis. Medical Image Analysis, 42:60 – 88, 2017.
  92. Geometry constrained weakly supervised object localization. In ECCV, 2020.
  93. A. Madabhushi. Digital pathology image analysis: opportunities and challenges. Imaging in Medicine, 1(1):7, 2009.
  94. Erasing integrated learning: A simple yet effective approach for weakly supervised object localization. In CVPR, 2020.
  95. Convolutional stn for weakly supervised object localization and beyond. In ICPR, 2020.
  96. Mrf-ann: a machine learning approach for automated er scoring of breast cancer immunohistochemical images. Journal of Microscopy, 267(2):117–129, 2017.
  97. Definitions, methods, and applications in interpretable machine learning. Proceedings of the National Academy of Sciences, 116(44):22071–22080, 2019.
  98. Constrained sampling for class-agnostic weakly supervised object localization. In Montreal AI symposium, 2022.
  99. Discriminative sampling of proposals in self-supervised transformers for weakly supervised object localization. In WACV workshop, 2023.
  100. R. Naidu and J. Michael. SS-CAM: smoothed score-cam for sharper visual feature localization. CoRR, abs/2006.14255, 2020.
  101. IS-CAM: integrated score-cam for axiomatic-based explanations. CoRR, abs/2010.03023, 2020.
  102. Automated gland segmentation and gleason grading of prostate histology by integrating low-, high-level and domain specific information. In Workshop on Microscopic Image Analysis with Applications in Biology, 2007.
  103. V. Nair and G.E. Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010.
  104. Smooth grad-cam++: An enhanced inference level visualization technique for deep convolutional neural network models. CoRR, abs/1908.01224, 2019.
  105. Is object localization for free?-weakly-supervised learning with convolutional neural networks. In CVPR, 2015.
  106. N. Otsu. A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man, and Cybernetics, 9(1):62–66, 1979.
  107. Constrained convolutional neural networks for weakly supervised segmentation. In ICCV, 2015.
  108. RISE: randomized input sampling for explanation of black-box models. In BMVC, 2018.
  109. V. Petsiuk et al. Black-box explanation of object detectors via saliency maps. CoRR, abs/2006.03204, 2020.
  110. Large-scale computations on histology images reveal grade-differentiating parameters for breast cancer. BMC medical imaging, 6(1):14, 2006.
  111. P. O. Pinheiro and R. Collobert. From image-level to pixel-level labeling with convolutional networks. In CVPR, 2015.
  112. Multiple-instance learning for medical image and video analysis. IEEE reviews in Biomedical Engineering, 10:213–234, 2017.
  113. Adaptive discriminant wavelet packet transform and local binary patterns for meningioma subtype classification. In MICCAI, 2008.
  114. Pairwise similarity knowledge transfer for weakly supervised object localization. In ECCV, 2020.
  115. You only look once: Unified, real-time object detection. In CVPR, 2016.
  116. Faster R-CNN: towards real-time object detection with region proposal networks. In NeurIPS, 2015.
  117. "why should i trust you?": Explaining the predictions of any classifier. In ACM SIGKDD 2016, 2016.
  118. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
  119. Mitosis detection in breast cancer histological images an icpr 2012 contest. Journal of Pathology Informatics, 4, 2013.
  120. Visual interpretability in 3d brain tumor segmentation network. Comput. Biol. Medicine, 133:104410, 2021.
  121. Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, volume 11700 of Lecture Notes in Computer Science. Springer, 2019.
  122. Toward interpretable machine learning: Transparent deep neural networks and beyond. CoRR, abs/2003.07631, 2020.
  123. Deep multiscale convolutional feature learning for weakly supervised localization of chest pathologies in x-ray images. In Intern. Workshop on Machine Learning in Medical Imaging, 2018.
  124. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, 2017.
  125. Deep learning assessment of tumor proliferation in breast cancer histological images. In Int. Conf. on Bioinformatics and Biomedicine, 2017.
  126. Automatic labeling of molecular biomarkers of whole slide immunohistochemistry images using fully convolutional networks. CoRR, abs/1612.09420, 2016.
  127. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, ICLR, 2015.
  128. Hide-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization. In ICCV, 2017.
  129. Gland segmentation in colon histology images: The glas challenge contest. Medical Image Analysis, 35:489–502, 2017.
  130. Learning from noisy labels with deep neural networks: A survey. CoRR, abs/2007.08199, 2020.
  131. Breast cancer histopathological image classification using convolutional neural networks. In International Joint Conference on Neural Network, 2016a.
  132. A dataset for breast cancer histopathological image classification. IEEE Transactions on Biomedical Engineering, 63(7):1455–1462, 2016b.
  133. Deep neural network models for computational histopathology: A survey. CoRR, abs/1912.12378, 2019.
  134. Dropout: A simple way to prevent neural networks from overfitting. JMLR, 15(56):1929–1958, 2014.
  135. Scorenet: Learning non-uniform attention and augmentation for transformer-based histopathological image classification. CoRR, abs/2202.07570, 2022.
  136. Multiple instance learning for histopathological breast cancer image classification. Expert Systems with Applications, 117:103 – 111, 2019.
  137. Training convolutional networks with noisy labels. coRR, abs/1406.2080, 2014.
  138. Pronet: Learning to propose object-specific boxes for cascaded neural networks. In CVPR, 2016.
  139. Rethinking the inception architecture for computer vision. In CVPR, 2016.
  140. Multifeature prostate cancer diagnosis and gleason grading of histological images. IEEE transactions on Medical Imaging, 26(10):1366–1378, 2007.
  141. Computer-aided detection and diagnosis of breast cancer with mammography: recent advances. IEEE Transactions on Information Technology in Biomedicine, 13(2):236–251, 2009.
  142. Normalized Cut Loss for Weakly-supervised CNN Segmentation. In CVPR, 2018.
  143. Automatic discovery of clinically interpretable imaging biomarkers for mycobacterium tuberculosis supersusceptibility using deep learning. EBioMedicine, 62:103094, 2020.
  144. Attention networks for weakly supervised object localization. In BMVC, 2016.
  145. Training data-efficient image transformers &; distillation through attention. In ICML, 2021.
  146. Modeling visual attention via selective tuning. Artificial Intelligence, 78(1):507–545, 1995. Special Volume on Computer Vision.
  147. Saliencymix: A saliency guided data augmentation strategy for better regularization. In ICLR, 2021.
  148. Breast cancer histopathology image analysis: A review. IEEE Transactions on Biomedical Engineering, 61(5):1400–1411, 2014.
  149. Min-entropy latent model for weakly supervised object detection. In CVPR, 2018.
  150. Exploring automatic prostate histopathology image gleason grading via local structure modeling. In Int. Conf. the IEEE Engineering in Medicine and Biology Society, 2015.
  151. Score-cam: Score-weighted visual explanations for convolutional neural networks. In CVPR workshop, 2020.
  152. Revisiting multiple instance neural networks. Pattern Recognition, 74:15–24, 2018.
  153. Shallow feature matters for weakly supervised object localization. In CVPR, 2021.
  154. Object region mining with adversarial erasing: A simple classification to semantic segmentation approach. In CVPR, 2017.
  155. Revisiting dilated convolution: A simple approach for weakly- and semi-supervised semantic segmentation. In CVPR, 2018.
  156. Invasive carcinomas and fibroadenomas of the breast: comparison of microvessel distributions–implications for imaging modalities. Radiology, 208(2):477–483, 1998.
  157. Deep learning based analysis of histopathological images of breast cancer. Frontiers in Genetics, 10, 2019.
  158. A deep convolutional neural network for segmenting and classifying epithelial and stromal regions in histopathological images. Neurocomputing, 191:214–223, 2016.
  159. Danet: Divergent activation for weakly supervised object localization. In ICCV, 2019.
  160. Combinational class activation maps for weakly supervised object localization. In WACV, 2020.
  161. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
  162. M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.
  163. Understanding deep learning requires rethinking generalization. CoRR, abs/1611.03530, 2017.
  164. Rethinking the route towards weakly supervised object localization. In CVPR, 2020a.
  165. mixup: Beyond empirical risk minimization. In ICLR, 2018a.
  166. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 126(10):1084–1102, 2018b.
  167. Adversarial complementary learning for weakly supervised object localization. In CVPR, 2018c.
  168. Self-produced guidance for weakly-supervised object localization. In ECCV, 2018d.
  169. Interpretable deep learning under fire. In USENIX Security Symposium, 2020b.
  170. Inter-image communication for weakly supervised localization. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, ECCV, Lecture Notes in Computer Science, 2020c.
  171. A survey on neural network interpretability. IEEE Trans. Emerg. Top. Comput. Intell., 5(5):726–742, 2021.
  172. Learning deep features for discriminative localization. In CVPR, 2016.
  173. Weakly supervised instance segmentation using class peak response. In CVPR, 2018.
  174. Z.-H. Zhou. Multi-instance learning: A survey. Department of Computer Science & Technology, Nanjing University, Tech. Rep, 2004.
  175. Z.-H. Zhou. A brief introduction to weakly supervised learning. National Science Review, 5(1):44–53, 2017.
  176. Soft proposal networks for weakly supervised object localization. In ICCV, 2017.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Jérôme Rony (14 papers)
  2. Soufiane Belharbi (31 papers)
  3. Jose Dolz (97 papers)
  4. Ismail Ben Ayed (133 papers)
  5. Luke McCaffrey (14 papers)
  6. Eric Granger (121 papers)
Citations (68)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now