School of Information Science and Engineering, Northeastern University, Shenyang 110819, China.
School of Information Science and Engineering, Northeastern University, Shenyang 110819, China.
J Neurosci Methods. 2024 Nov;411:110269. doi: 10.1016/j.jneumeth.2024.110269. Epub 2024 Aug 31.
Image reconstruction is a critical task in brain decoding research, primarily utilizing functional magnetic resonance imaging (fMRI) data. However, due to challenges such as limited samples in fMRI data, the quality of reconstruction results often remains poor.
We proposed a three-stage multi-level deep fusion model (TS-ML-DFM). The model employed a three-stage training process, encompassing components such as image encoders, generators, discriminators, and fMRI encoders. In this method, we incorporated distinct supplementary features derived separately from depth images and original images. Additionally, the method integrated several components, including a random shift module, dual attention module, and multi-level feature fusion module.
In both qualitative and quantitative comparisons on the Horikawa17 and VanGerven10 datasets, our method exhibited excellent performance.
For example, on the primary Horikawa17 dataset, our method was compared with other leading methods based on metrics the average hash value, histogram similarity, mutual information, structural similarity accuracy, AlexNet(2), AlexNet(5), and pairwise human perceptual similarity accuracy. Compared to the second-ranked results in each metric, the proposed method achieved improvements of 0.99 %, 3.62 %, 3.73 %, 2.45 %, 3.51 %, 0.62 %, and 1.03 %, respectively. In terms of the SwAV top-level semantic metric, a substantial improvement of 10.53 % was achieved compared to the second-ranked result in the pixel-level reconstruction methods.
The TS-ML-DFM method proposed in this study, when applied to decoding brain visual patterns using fMRI data, has outperformed previous algorithms, thereby facilitating further advancements in research within this field.
图像重建是脑解码研究中的一项关键任务，主要利用功能磁共振成像 (fMRI) 数据。然而，由于 fMRI 数据中样本有限等挑战，重建结果的质量往往仍然较差。
我们提出了一种三阶段多层次深度融合模型（TS-ML-DFM）。该模型采用三阶段训练过程，包括图像编码器、生成器、鉴别器和 fMRI 编码器等组件。在这种方法中，我们结合了从深度图像和原始图像分别提取的不同补充特征。此外，该方法集成了多个组件，包括随机移位模块、双注意力模块和多层次特征融合模块。
在 Horikawa17 和 VanGerven10 数据集上的定性和定量比较中，我们的方法表现出了优异的性能。
例如，在主要的 Horikawa17 数据集上，我们的方法与其他领先方法进行了比较，比较指标包括平均哈希值、直方图相似度、互信息、结构相似性精度、AlexNet(2)、AlexNet(5)和成对人类感知相似性精度。与每个指标中排名第二的结果相比，所提出的方法分别提高了 0.99%、3.62%、3.73%、2.45%、3.51%、0.62%和 1.03%。在 SwAV 顶级语义指标方面，与像素级重建方法中排名第二的结果相比，取得了 10.53%的显著提高。
本研究提出的 TS-ML-DFM 方法在使用 fMRI 数据解码大脑视觉模式方面优于以前的算法，从而促进了该领域研究的进一步发展。
