Computational Neuroscience Laboratories, Advanced Telecommunications Research Institute International, Kyoto, Japan.
Graduate school of Informatics, Kyoto University, Kyoto, Japan.
PLoS Comput Biol. 2019 Jan 14;15(1):e1006633. doi: 10.1371/journal.pcbi.1006633. eCollection 2019 Jan.
The mental contents of perception and imagery are thought to be encoded in hierarchical representations in the brain, but previous attempts to visualize perceptual contents have failed to capitalize on multiple levels of the hierarchy, leaving it challenging to reconstruct internal imagery. Recent work showed that visual cortical activity measured by functional magnetic resonance imaging (fMRI) can be decoded (translated) into the hierarchical features of a pre-trained deep neural network (DNN) for the same input image, providing a way to make use of the information from hierarchical visual features. Here, we present a novel image reconstruction method, in which the pixel values of an image are optimized to make its DNN features similar to those decoded from human brain activity at multiple layers. We found that our method was able to reliably produce reconstructions that resembled the viewed natural images. A natural image prior introduced by a deep generator neural network effectively rendered semantically meaningful details to the reconstructions. Human judgment of the reconstructions supported the effectiveness of combining multiple DNN layers to enhance the visual quality of generated images. While our model was solely trained with natural images, it successfully generalized to artificial shapes, indicating that our model was not simply matching to exemplars. The same analysis applied to mental imagery demonstrated rudimentary reconstructions of the subjective content. Our results suggest that our method can effectively combine hierarchical neural representations to reconstruct perceptual and subjective images, providing a new window into the internal contents of the brain.
感知和意象的心理内容被认为是在大脑中以层次化的方式进行编码的，但之前尝试可视化感知内容的尝试都未能充分利用层次结构的多个层次，因此难以重建内部意象。最近的研究表明，通过功能磁共振成像（fMRI）测量的视觉皮层活动可以被解码（翻译）为同一输入图像的预训练深度神经网络（DNN）的层次特征，为利用层次化视觉特征的信息提供了一种方法。在这里，我们提出了一种新的图像重建方法，其中图像的像素值被优化，以使 DNN 特征与从人类大脑活动在多个层次上解码的特征相似。我们发现，我们的方法能够可靠地生成与所观看的自然图像相似的重建图像。通过深度生成器神经网络引入的自然图像先验有效地为重建图像渲染了语义上有意义的细节。人类对重建图像的判断支持了结合多个 DNN 层来提高生成图像的视觉质量的有效性。虽然我们的模型仅用自然图像进行训练，但它成功地推广到了人工形状，这表明我们的模型不是简单地与范例匹配。同样的分析应用于心理意象，证明了对主观内容的初步重建。我们的结果表明，我们的方法可以有效地结合层次化的神经表示来重建感知和主观图像，为大脑的内部内容提供了一个新的窗口。
