Key Laboratory of Child Development and Learning Science of Ministry of Education, School of Biological Science & Medical Engineering, Southeast University, Nanjing, Jiangsu 210096, China.
School of Innovations, Sanjiang University, China; School of Electronic Science and Engineering, Nanjing University, China.
J Neurosci Methods. 2024 Dec;412:110292. doi: 10.1016/j.jneumeth.2024.110292. Epub 2024 Sep 17.
Due to the sparse encoding character of the human visual cortex and the scarcity of paired training samples for {images, fMRIs}, voxel selection is an effective means of reconstructing perceived images from fMRI. However, the existing data-driven voxel selection methods have not achieved satisfactory results.
Here, a novel deep reinforcement learning-guided sparse voxel (DRL-SV) decoding model is proposed to reconstruct perceived images from fMRI. We innovatively describe voxel selection as a Markov decision process (MDP), training agents to select voxels that are highly involved in specific visual encoding.
Experimental results on two public datasets verify the effectiveness of the proposed DRL-SV, which can accurately select voxels highly involved in neural encoding, thereby improving the quality of visual image reconstruction.
We qualitatively and quantitatively compared our results with the state-of-the-art (SOTA) methods, getting better reconstruction results. We compared the proposed DRL-SV with traditional data-driven baseline methods, obtaining sparser voxel selection results, but better reconstruction performance.
DRL-SV can accurately select voxels involved in visual encoding on few-shot, compared to data-driven voxel selection methods. The proposed decoding model provides a new avenue to improving the image reconstruction quality of the primary visual cortex.
由于人类视觉皮层的稀疏编码特性以及用于{图像、功能磁共振成像(fMRI)}的配对训练样本稀缺，体素选择是从 fMRI 中重建感知图像的有效手段。然而，现有的数据驱动体素选择方法尚未取得令人满意的效果。
本文提出了一种新颖的基于深度强化学习引导稀疏体素（DRL-SV）解码模型，用于从 fMRI 中重建感知图像。我们创新性地将体素选择描述为马尔可夫决策过程（MDP），训练代理选择与特定视觉编码高度相关的体素。
实验结果在两个公共数据集上验证了所提出的 DRL-SV 的有效性，该方法可以准确地选择高度参与神经编码的体素，从而提高视觉图像重建的质量。
我们对结果进行了定性和定量的比较，与最先进的(SOTA)方法相比，获得了更好的重建结果。我们将提出的 DRL-SV 与传统的数据驱动基线方法进行了比较，虽然体素选择结果更稀疏，但重建性能更好。
与数据驱动的体素选择方法相比，DRL-SV 可以在样本数量较少的情况下准确地选择参与视觉编码的体素。所提出的解码模型为提高初级视觉皮层的图像重建质量提供了新途径。
