Self-Supervised Super-Resolution Approach for Isotropic Reconstruction of 3D Electron Microscopy Images from Anisotropic Acquisition (2309.10646v1)
Abstract: Three-dimensional electron microscopy (3DEM) is an essential technique to investigate volumetric tissue ultra-structure. Due to technical limitations and high imaging costs, samples are often imaged anisotropically, where resolution in the axial direction ($z$) is lower than in the lateral directions $(x,y)$. This anisotropy 3DEM can hamper subsequent analysis and visualization tasks. To overcome this limitation, we propose a novel deep-learning (DL)-based self-supervised super-resolution approach that computationally reconstructs isotropic 3DEM from the anisotropic acquisition. The proposed DL-based framework is built upon the U-shape architecture incorporating vision-transformer (ViT) blocks, enabling high-capability learning of local and global multi-scale image dependencies. To train the tailored network, we employ a self-supervised approach. Specifically, we generate pairs of anisotropic and isotropic training datasets from the given anisotropic 3DEM data. By feeding the given anisotropic 3DEM dataset in the trained network through our proposed framework, the isotropic 3DEM is obtained. Importantly, this isotropic reconstruction approach relies solely on the given anisotropic 3DEM dataset and does not require pairs of co-registered anisotropic and isotropic 3DEM training datasets. To evaluate the effectiveness of the proposed method, we conducted experiments using three 3DEM datasets acquired from brain. The experimental results demonstrated that our proposed framework could successfully reconstruct isotropic 3DEM from the anisotropic acquisition.
- S. Mikula, “Progress towards mammalian whole-brain cellular connectomics,” Frontiers in neuroanatomy, vol. 10, p. 62, 2016.
- P. Song, X. Deng, J. F. Mota, N. Deligiannis, P. L. Dragotti, and M. R. Rodrigues, “Multimodal image super-resolution via joint sparse representations induced by coupled dictionaries,” IEEE Transactions on Computational Imaging, vol. 6, pp. 57–72, 2019.
- J. Yang, J. Wright, T. S. Huang, and Y. Ma, “Image super-resolution via sparse representation,” IEEE transactions on image processing, vol. 19, no. 11, pp. 2861–2873, 2010.
- T. Hu, J. Nunez-Iglesias, S. Vitaladevuni, L. Scheffer, S. Xu, M. Bolorizadeh, H. Hess, R. Fetter, and D. Chklovskii, “Super-resolution using sparse representations over learned dictionaries: Reconstruction of brain structure using electron microscopy,” arXiv preprint arXiv:1210.0564, 2012.
- R. A. Salo, I. Belevich, E. Manninen, E. Jokitalo, O. Gröhn, and A. Sierra, “Quantification of anisotropy and orientation in 3d electron microscopy and diffusion tensor imaging in injured rat brain,” Neuroimage, vol. 172, pp. 404–414, 2018.
- A. Abdollahzadeh, I. Belevich, E. Jokitalo, A. Sierra, and J. Tohka, “Deepacson automated segmentation of white matter in 3d electron microscopy,” Communications biology, vol. 4, no. 1, p. 179, 2021.
- H. Chen, X. He, L. Qing, Y. Wu, C. Ren, R. E. Sheriff, and C. Zhu, “Real-world single image super-resolution: A brief review,” Information Fusion, vol. 79, pp. 124–145, 2022.
- Y. Sui, O. Afacan, C. Jaimes, A. Gholipour, and S. K. Warfield, “Scan-specific generative neural network for mri super-resolution reconstruction,” IEEE Transactions on Medical Imaging, vol. 41, no. 6, pp. 1383–1399, 2022.
- L. Fang, F. Monroe, S. W. Novak, L. Kirk, C. R. Schiavon, S. B. Yu, T. Zhang, M. Wu, K. Kastner, A. A. Latif, et al., “Deep learning-based point-scanning super-resolution imaging,” Nature methods, vol. 18, no. 4, pp. 406–416, 2021.
- L. Heinrich, J. A. Bogovic, and S. Saalfeld, “Deep learning for isotropic super-resolution from non-isotropic 3d electron microscopy,” in Medical Image Computing and Computer-Assisted Intervention- MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part II 20, pp. 135–143, Springer, 2017.
- A. Shocher, N. Cohen, and M. Irani, ““zero-shot” super-resolution using deep internal learning,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3118–3126, 2018.
- M. Weigert, L. Royer, F. Jug, and G. Myers, “Isotropic reconstruction of 3d fluorescence microscopy images using convolutional neural networks,” in Medical Image Computing and Computer-Assisted Intervention- MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part II 20, pp. 126–134, Springer, 2017.
- C. Zhao, B. E. Dewey, D. L. Pham, P. A. Calabresi, D. S. Reich, and J. L. Prince, “Smore: a self-supervised anti-aliasing and super-resolution algorithm for mri using deep learning,” IEEE transactions on medical imaging, vol. 40, no. 3, pp. 805–817, 2020.
- Z. Wang, X. Cun, J. Bao, W. Zhou, J. Liu, and H. Li, “Uformer: A general u-shaped transformer for image restoration,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17683–17693, 2022.
- J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte, “Swinir: Image restoration using swin transformer,” in Proceedings of the IEEE/CVF international conference on computer vision, pp. 1833–1844, 2021.
- Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012–10022, 2021.
- M. Delbracio, H. Talebi, and P. Milanfar, “Projected distribution loss for image enhancement,” arXiv preprint arXiv:2012.09289, 2020.
- D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
- https://www.epfl.ch/labs/cvlab/data/data-em/.
- K. Zhang, W. Zuo, and L. Zhang, “Learning a single convolutional super-resolution network for multiple degradations,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3262–3271, 2018.
- Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” IEEE transactions on image processing, vol. 13, no. 4, pp. 600–612, 2004.