Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Reconstructing Retinal Visual Images from 3T fMRI Data Enhanced by Unsupervised Learning (2404.05107v1)

Published 7 Apr 2024 in cs.CV

Abstract: The reconstruction of human visual inputs from brain activity, particularly through functional Magnetic Resonance Imaging (fMRI), holds promising avenues for unraveling the mechanisms of the human visual system. Despite the significant strides made by deep learning methods in improving the quality and interpretability of visual reconstruction, there remains a substantial demand for high-quality, long-duration, subject-specific 7-Tesla fMRI experiments. The challenge arises in integrating diverse smaller 3-Tesla datasets or accommodating new subjects with brief and low-quality fMRI scans. In response to these constraints, we propose a novel framework that generates enhanced 3T fMRI data through an unsupervised Generative Adversarial Network (GAN), leveraging unpaired training across two distinct fMRI datasets in 7T and 3T, respectively. This approach aims to overcome the limitations of the scarcity of high-quality 7-Tesla data and the challenges associated with brief and low-quality scans in 3-Tesla experiments. In this paper, we demonstrate the reconstruction capabilities of the enhanced 3T fMRI data, highlighting its proficiency in generating superior input visual images compared to data-intensive methods trained and tested on a single subject.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (25)
  1. Fang et al., “Reconstructing perceptive images from brain activity by shape-semantic gan,” Advances in Neural Information Processing Systems, vol. 33, pp. 13038–13048, 2020.
  2. Ren et al., “Reconstructing seen image from brain activity by visually-guided cognitive representation and adversarial learning,” NeuroImage, vol. 228, pp. 117602, 2021.
  3. Le et al., “Brain2pix: Fully convolutional naturalistic video reconstruction from brain activity,” BioRxiv, pp. 2021–02, 2021.
  4. Seeliger et al., “Generative adversarial networks for reconstructing natural images from brain activity,” NeuroImage, vol. 181, pp. 775–785, 2018.
  5. Parthasarathy et al., “Neural networks for efficient bayesian decoding of natural images from retinal neurons,” Advances in Neural Information Processing Systems, vol. 30, 2017.
  6. Gaziv et al., “Self-supervised natural image reconstruction and large-scale semantic classification from brain activity,” NeuroImage, vol. 254, pp. 119121, 2022.
  7. Chang et al., “Bold5000, a public fmri dataset while viewing 5000 visual images,” Scientific data, vol. 6, no. 1, pp. 49, 2019.
  8. Shen et al., “Deep image reconstruction from human brain activity,” PLoS computational biology, vol. 15, no. 1, pp. e1006633, 2019.
  9. Horikawa and Kamitani, “Generic decoding of seen and imagined objects using hierarchical visual features,” Nature communications, vol. 8, no. 1, pp. 15037, 2017.
  10. Allen et al., “A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence,” Nature neuroscience, vol. 25, no. 1, pp. 116–126, 2022.
  11. Gong et al., “A large-scale fmri dataset for the visual processing of naturalistic scenes,” Scientific Data, vol. 10, no. 1, pp. 559, 2023.
  12. Kay et al., “Identifying natural images from human brain activity,” Nature, vol. 452, no. 7185, pp. 352–355, 2008.
  13. Naselaris et al., “Bayesian reconstruction of natural images from human brain activity,” Neuron, vol. 63, no. 6, pp. 902–915, 2009.
  14. Chen et al., “Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 22710–22720.
  15. Lin et al., “Mind reader: Reconstructing complex images from brain activities,” Advances in Neural Information Processing Systems, vol. 35, pp. 29624–29636, 2022.
  16. Takagi and Nishimoto, “High-resolution image reconstruction with latent diffusion models from human brain activity,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 14453–14463.
  17. Wang et al., “Optimal transport for unsupervised denoising learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 2, pp. 2104–2118, 2022.
  18. Zhu et al., “Optimal transport guided unsupervised learning for enhancing low-quality retinal images,” Proc IEEE Int Symp Biomed Imaging, vol. 2023, Apr 2023.
  19. Heusel et al., “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” Advances in neural information processing systems, vol. 30, 2017.
  20. Rombach et al., “High-resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10684–10695.
  21. Lin et al., “Microsoft coco: Common objects in context,” in Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer, 2014, pp. 740–755.
  22. Dickie et al., “Ciftify: A framework for surface-based analysis of legacy mr acquisitions,” Neuroimage, vol. 197, pp. 818–826, 2019.
  23. Markello et al., “Neuromaps: structural and functional interpretation of brain maps,” Nature Methods, vol. 19, no. 11, pp. 1472–1479, 2022.
  24. Ledig et al., “Photo-realistic single image super-resolution using a generative adversarial network,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4681–4690.
  25. Ho et al., “Denoising diffusion probabilistic models,” Advances in neural information processing systems, vol. 33, pp. 6840–6851, 2020.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now