Zhang Yijun, Bu Tong, Zhang Jiyuan, Tang Shiming, Yu Zhaofei, Liu Jian K, Huang Tiejun
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai 200240.
Department of Computer Science and Technology, Peking University, Peking 100871, P.R.C.
Neural Comput. 2022 May 19;34(6):1369-1397. doi: 10.1162/neco_a_01498.
Images of visual scenes comprise essential features important for visual cognition of the brain. The complexity of visual features lies at different levels, from simple artificial patterns to natural images with different scenes. It has been a focus of using stimulus images to predict neural responses. However, it remains unclear how to extract features from neuronal responses. Here we address this question by leveraging two-photon calcium neural data recorded from the visual cortex of awake macaque monkeys. With stimuli including various categories of artificial patterns and diverse scenes of natural images, we employed a deep neural network decoder inspired by image segmentation technique. Consistent with the notation of sparse coding for natural images, a few neurons with stronger responses dominated the decoding performance, whereas decoding of ar tificial patterns needs a large number of neurons. When natural images using the model pretrained on artificial patterns are decoded, salient features of natural scenes can be extracted, as well as the conventional category information. Altogether, our results give a new perspective on studying neural encoding principles using reverse-engineering decoding strategies.
视觉场景图像包含对大脑视觉认知至关重要的特征。视觉特征的复杂性存在于不同层面，从简单的人工图案到具有不同场景的自然图像。利用刺激图像来预测神经反应一直是一个研究重点。然而，如何从神经元反应中提取特征仍不清楚。在这里，我们通过利用从清醒猕猴视觉皮层记录的双光子钙神经数据来解决这个问题。对于包括各类人工图案和不同场景自然图像的刺激，我们采用了受图像分割技术启发的深度神经网络解码器。与自然图像的稀疏编码概念一致，少数反应较强的神经元主导了解码性能，而人工图案的解码则需要大量神经元。当对使用在人工图案上预训练的模型的自然图像进行解码时，不仅可以提取自然场景的显著特征，还能提取传统的类别信息。总之，我们的结果为使用逆向工程解码策略研究神经编码原理提供了一个新视角。
