Neuroscience Program, Bates College, Lewiston, Maine 04240
Department of Psychological & Brain Sciences, Neuroscience Program, Colgate University, Hamilton, New York 13346.
J Neurosci. 2020 Jul 1;40(27):5283-5299. doi: 10.1523/JNEUROSCI.2088-19.2020. Epub 2020 May 28.
Human scene categorization is characterized by its remarkable speed. While many visual and conceptual features have been linked to this ability, significant correlations exist between feature spaces, impeding our ability to determine their relative contributions to scene categorization. Here, we used a whitening transformation to decorrelate a variety of visual and conceptual features and assess the time course of their unique contributions to scene categorization. Participants (both sexes) viewed 2250 full-color scene images drawn from 30 different scene categories while having their brain activity measured through 256-channel EEG. We examined the variance explained at each electrode and time point of visual event-related potential (vERP) data from nine different whitened encoding models. These ranged from low-level features obtained from filter outputs to high-level conceptual features requiring human annotation. The amount of category information in the vERPs was assessed through multivariate decoding methods. Behavioral similarity measures were obtained in separate crowdsourced experiments. We found that all nine models together contributed 78% of the variance of human scene similarity assessments and were within the noise ceiling of the vERP data. Low-level models explained earlier vERP variability (88 ms after image onset), whereas high-level models explained later variance (169 ms). Critically, only high-level models shared vERP variability with behavior. Together, these results suggest that scene categorization is primarily a high-level process, but reliant on previously extracted low-level features. In a single fixation, we glean enough information to describe a general scene category. Many types of features are associated with scene categories, ranging from low-level properties, such as colors and contours, to high-level properties, such as objects and attributes. Because these properties are correlated, it is difficult to understand each property's unique contributions to scene categorization. This work uses a whitening transformation to remove the correlations between features and examines the extent to which each feature contributes to visual event-related potentials over time. We found that low-level visual features contributed first but were not correlated with categorization behavior. High-level features followed 80 ms later, providing key insights into how the brain makes sense of a complex visual world.
人类场景分类的特点是速度极快。虽然许多视觉和概念特征都与这种能力有关，但特征空间之间存在显著的相关性，这阻碍了我们确定它们对场景分类相对贡献的能力。在这里，我们使用白化变换来解相关各种视觉和概念特征，并评估它们对场景分类的独特贡献的时间进程。参与者（男女皆有）观看了 2250 张来自 30 个不同场景类别的全彩色场景图像，同时通过 256 通道 EEG 测量他们的大脑活动。我们检查了从九个不同白化编码模型获得的视觉事件相关电位（vERP）数据的每个电极和时间点的方差解释。这些模型从滤波器输出获得的低水平特征到需要人类注释的高水平概念特征。通过多元解码方法评估 vERP 中的类别信息量。在单独的众包实验中获得行为相似性度量。我们发现，所有九个模型共同贡献了 78%的人类场景相似性评估的方差，并且处于 vERP 数据的噪声上限内。低水平模型解释了更早的 vERP 可变性（图像出现后 88 毫秒），而高水平模型解释了较晚的方差（169 毫秒）。关键是，只有高水平模型与行为共享 vERP 可变性。总的来说，这些结果表明场景分类主要是一个高水平的过程，但依赖于先前提取的低水平特征。在一个单一的注视中，我们获得了足够的信息来描述一个一般的场景类别。与场景类别相关的特征有很多种，从颜色和轮廓等低水平属性到物体和属性等高水平属性。由于这些属性是相关的，因此很难理解每个属性对场景分类的独特贡献。这项工作使用白化变换来消除特征之间的相关性，并检查每个特征在时间上对视觉事件相关电位的贡献程度。我们发现，低水平视觉特征首先贡献，但与分类行为不相关。高水平特征在 80 毫秒后出现，为大脑如何理解复杂的视觉世界提供了关键见解。
