Brain-aligning of semantic vectors improves neural decoding of visual stimuli (2403.15176v3)

Abstract: The development of algorithms to accurately decode of neural information is a long-standing effort in the field of neuroscience. Brain decoding is typically employed by training machine learning models to map neural data onto a preestablished vector representation of stimulus features. These vectors are usually derived from image- and/or text-based feature spaces. Nonetheless, the intrinsic characteristics of these vectors might be fundamentally different than those encoded by the brain, limiting the ability of algorithms to accurately learn this mapping. To address this issue, here, we propose a representation learning framework, called brain-aligning of semantic vectors, that fine-tunes pretrained feature vectors to better align with the structure of neural representations of visual stimuli in the human brain. We trained this model with functional magnetic resonance imaging (fMRI) data representing 150 visual stimulus categories; then, we performed zero-shot brain decoding on 1) fMRI, 2) magnetoencephalography (MEG), and 3) electrocorticography (ECoG) data reflecting neural representations of visual stimuli. By using fMRI-based brain-aligned vectors, the zero-shot decoding accuracy all three neuroimaging datasets increased. This finding underscores the potential of leveraging a richer array of brainderived features to increase the performance of brain decoding algorithms.