School of Electronic Engineering, Xidian University, Xi'an 710071, China.
Group 42 (G42), Abu Dhabi, United Arab Emirates.
Neural Netw. 2023 Sep;166:487-500. doi: 10.1016/j.neunet.2023.07.016. Epub 2023 Jul 28.
Reconstructing visual experience from brain responses measured by functional magnetic resonance imaging (fMRI) is a challenging yet important research topic in brain decoding, especially it has proved more difficult to decode visually similar stimuli, such as faces. Although face attributes are known as the key to face recognition, most existing methods generally ignore how to decode facial attributes more precisely in perceived face reconstruction, which often leads to indistinguishable reconstructed faces. To solve this problem, we propose a novel neural decoding framework called VSPnet (voxel2style2pixel) by establishing hierarchical encoding and decoding networks with disentangled latent representations as media, so that to recover visual stimuli more elaborately. And we design a hierarchical visual encoder (named HVE) to pre-extract features containing both high-level semantic knowledge and low-level visual details from stimuli. The proposed VSPnet consists of two networks: Multi-branch cognitive encoder and style-based image generator. The encoder network is constructed by multiple linear regression branches to map brain signals to the latent space provided by the pre-extracted visual features and obtain representations containing hierarchical information consistent to the corresponding stimuli. We make the generator network inspired by StyleGAN to untangle the complexity of fMRI representations and generate images. And the HVE network is composed of a standard feature pyramid over a ResNet backbone. Extensive experimental results on the latest public datasets have demonstrated the reconstruction accuracy of our proposed method outperforms the state-of-the-art approaches and the identifiability of different reconstructed faces has been greatly improved. In particular, we achieve feature editing for several facial attributes in fMRI domain based on the multiview (i.e., visual stimuli and evoked fMRI) latent representations.
从功能磁共振成像 (fMRI) 测量的大脑反应中重建视觉体验是脑解码中的一个具有挑战性但很重要的研究课题，特别是在解码视觉上相似的刺激（如人脸）时，这已经被证明更加困难。尽管面部属性是人脸识别的关键，但大多数现有的方法通常忽略了如何在感知的人脸重建中更精确地解码面部属性，这往往导致重建的人脸无法区分。为了解决这个问题，我们提出了一种名为 VSPnet（voxel2style2pixel）的新型神经解码框架，通过建立具有分离潜在表示的分层编码和解码网络作为媒介，从而更精细地恢复视觉刺激。我们设计了一个分层视觉编码器（命名为 HVE），从刺激中预先提取包含高层语义知识和底层视觉细节的特征。所提出的 VSPnet 由两个网络组成：多分支认知编码器和基于风格的图像生成器。编码器网络由多个线性回归分支构建，将大脑信号映射到由预先提取的视觉特征提供的潜在空间，并获得包含与相应刺激一致的分层信息的表示。我们受 StyleGAN 的启发构建了生成器网络，以解开 fMRI 表示的复杂性并生成图像。并且 HVE 网络由一个基于 ResNet 骨干的标准特征金字塔组成。在最新的公共数据集上进行的广泛实验结果表明，我们提出的方法的重建准确性优于最先进的方法，并且不同重建人脸的可识别性得到了大大提高。特别是，我们基于多视图（即视觉刺激和诱发 fMRI）潜在表示，在 fMRI 域中实现了几个面部属性的特征编辑。
