Wang Yun
Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai 200433, China.
Key Laboratory of Computational Neuroscience and Brain-Inspired Intelligence, Fudan University, Ministry of Education, Shanghai 200433, China.
Brain Sci. 2024 Sep 28;14(10):988. doi: 10.3390/brainsci14100988.
BACKGROUND/OBJECTIVES: Neural decoding methods are often limited by the performance of brain encoders, which map complex brain signals into a latent representation space of perception information. These brain encoders are constrained by the limited amount of paired brain and stimuli data available for training, making it challenging to learn rich neural representations.
To address this limitation, we present a novel multimodal training approach using paired image and functional magnetic resonance imaging (fMRI) data to establish a brain masked autoencoder that learns the interactions between images and brain activities. Subsequently, we employ a diffusion model conditioned on brain data to decode realistic images.
Our method achieves high-quality decoding results in semantic contents and low-level visual attributes, outperforming previous methods both qualitatively and quantitatively, while maintaining computational efficiency. Additionally, our method is applied to decode artificial patterns across region of interests (ROIs) to explore their functional properties. We not only validate existing knowledge concerning ROIs but also unveil new insights, such as the synergy between early visual cortex and higher-level scene ROIs, as well as the competition within the higher-level scene ROIs.
These findings provide valuable insights for future directions in the field of neural decoding.
背景/目的：神经解码方法通常受大脑编码器性能的限制，大脑编码器将复杂的大脑信号映射到感知信息的潜在表示空间。这些大脑编码器受到可用于训练的配对大脑和刺激数据量有限的约束，使得学习丰富的神经表示具有挑战性。
为了解决这一限制，我们提出了一种新颖的多模态训练方法，使用配对的图像和功能磁共振成像（fMRI）数据来建立一个大脑掩码自动编码器，以学习图像与大脑活动之间的相互作用。随后，我们采用基于大脑数据的扩散模型来解码逼真的图像。
我们的方法在语义内容和低级视觉属性方面实现了高质量的解码结果，在定性和定量方面均优于先前的方法，同时保持了计算效率。此外，我们的方法应用于跨感兴趣区域（ROI）解码人工模式，以探索其功能特性。我们不仅验证了关于ROI的现有知识，还揭示了新的见解，例如早期视觉皮层与高级场景ROI之间的协同作用，以及高级场景ROI内的竞争。
这些发现为神经解码领域的未来方向提供了有价值的见解。
