Wakita Suguru, Orima Taiki, Motoyoshi Isamu
Department of Life Sciences, The University of Tokyo, Tokyo, Japan.
Japan Society for the Promotion of Science, Tokyo, Japan.
Front Comput Neurosci. 2021 Nov 19;15:754587. doi: 10.3389/fncom.2021.754587. eCollection 2021.
Recent advances in brain decoding have made it possible to classify image categories based on neural activity. Increasing numbers of studies have further attempted to reconstruct the image itself. However, because images of objects and scenes inherently involve spatial layout information, the reconstruction usually requires retinotopically organized neural data with high spatial resolution, such as fMRI signals. In contrast, spatial layout does not matter in the perception of "texture," which is known to be represented as spatially global image statistics in the visual cortex. This property of "texture" enables us to reconstruct the perceived image from EEG signals, which have a low spatial resolution. Here, we propose an MVAE-based approach for reconstructing texture images from visual evoked potentials measured from observers viewing natural textures such as the textures of various surfaces and object ensembles. This approach allowed us to reconstruct images that perceptually resemble the original textures with a photographic appearance. The present approach can be used as a method for decoding the highly detailed "impression" of sensory stimuli from brain activity.
脑解码技术的最新进展使得基于神经活动对图像类别进行分类成为可能。越来越多的研究进一步尝试重建图像本身。然而，由于物体和场景的图像本质上涉及空间布局信息，重建通常需要具有高空间分辨率的视网膜拓扑组织神经数据，如功能磁共振成像（fMRI）信号。相比之下，空间布局在“纹理”感知中并不重要，已知“纹理”在视觉皮层中以空间全局图像统计的形式表示。“纹理”的这一特性使我们能够从具有低空间分辨率的脑电图（EEG）信号中重建感知到的图像。在这里，我们提出了一种基于变分自编码器（MVAE）的方法，用于从观察者观看自然纹理（如各种表面和物体集合的纹理）时测量的视觉诱发电位中重建纹理图像。这种方法使我们能够重建出在感知上与原始纹理相似且具有照片外观的图像。本方法可作为一种从脑活动中解码感官刺激的高度详细“印象”的方法。
