Department of Neuroscience, University of Rochester, Rochester, New York 14642
Del Monte Institute for Neuroscience, University of Rochester, Rochester, New York 14642.
J Neurosci. 2021 May 5;41(18):4100-4119. doi: 10.1523/JNEUROSCI.1152-20.2021. Epub 2021 Mar 22.
Understanding how and where in the brain sentence-level meaning is constructed from words presents a major scientific challenge. Recent advances have begun to explain brain activation elicited by sentences using vector models of word meaning derived from patterns of word co-occurrence in text corpora. These studies have helped map out semantic representation across a distributed brain network spanning temporal, parietal, and frontal cortex. However, it remains unclear whether activation patterns within regions reflect unified representations of sentence-level meaning, as opposed to superpositions of context-independent component words. This is because models have typically represented sentences as "bags-of-words" that neglect sentence-level structure. To address this issue, we interrogated fMRI activation elicited as 240 sentences were read by 14 participants (9 female, 5 male), using sentences encoded by a recurrent deep artificial neural-network trained on a sentence inference task (InferSent). Recurrent connections and nonlinear filters enable InferSent to transform sequences of word vectors into unified "propositional" sentence representations suitable for evaluating intersentence entailment relations. Using voxelwise encoding modeling, we demonstrate that InferSent predicts elements of fMRI activation that cannot be predicted by bag-of-words models and sentence models using grammatical rules to assemble word vectors. This effect occurs throughout a distributed network, which suggests that propositional sentence-level meaning is represented within and across multiple cortical regions rather than at any single site. In follow-up analyses, we place results in the context of other deep network approaches (ELMo and BERT) and estimate the degree of unpredicted neural signal using an "experiential" semantic model and cross-participant encoding. A modern-day scientific challenge is to understand how the human brain transforms word sequences into representations of sentence meaning. A recent approach, emerging from advances in functional neuroimaging, big data, and machine learning, is to computationally model meaning, and use models to predict brain activity. Such models have helped map a cortical semantic information-processing network. However, how unified sentence-level information, as opposed to word-level units, is represented throughout this network remains unclear. This is because models have typically represented sentences as unordered "bags-of-words." Using a deep artificial neural network that recurrently and nonlinearly combines word representations into unified propositional sentence representations, we provide evidence that sentence-level information is encoded throughout a cortical network, rather than in a single region.
理解大脑如何以及在何处构建句子级别的意义是一个重大的科学挑战。最近的进展开始使用从文本语料库中的单词共现模式得出的单词意义向量模型来解释句子引发的大脑激活。这些研究有助于在跨越颞叶、顶叶和额叶皮层的分布式大脑网络中描绘语义表示。然而，目前尚不清楚区域内的激活模式是否反映了句子级意义的统一表示，而不是独立于上下文的组成词的叠加。这是因为模型通常将句子表示为“单词袋”，从而忽略了句子级结构。为了解决这个问题，我们使用 14 名参与者（9 名女性，5 名男性）阅读的 240 个句子，对 fMRI 激活进行了询问，这些句子是由经过句子推理任务（InferSent）训练的递归深度人工神经网络编码的。递归连接和非线性滤波器使 InferSent 能够将单词向量序列转换为适合评估句子蕴涵关系的统一“命题”句子表示。使用体素编码建模，我们证明 InferSent 可以预测单词袋模型和使用语法规则组装单词向量的句子模型无法预测的 fMRI 激活元素。这种效应发生在整个分布式网络中，这表明命题句子级别的意义是在多个皮质区域内和跨区域表示的，而不是在任何单个位置。在后续分析中，我们将结果置于其他深度网络方法（ELMo 和 BERT）的背景下，并使用“经验”语义模型和跨参与者编码来估计未预测的神经信号的程度。现代科学挑战是理解人类大脑如何将单词序列转换为句子意义的表示。一种新出现的方法，源自功能神经影像学、大数据和机器学习方面的进展，是对意义进行计算建模，并使用模型来预测大脑活动。这种模型帮助绘制了皮质语义信息处理网络。然而，整个网络中如何表示统一的句子级信息，而不是词级单元，仍然不清楚。这是因为模型通常将句子表示为无序的“单词袋”。我们使用深度人工神经网络，该神经网络递归且非线性地将单词表示组合成统一的命题句子表示，提供了证据表明句子级信息是在皮质网络中编码的，而不是在单个区域中编码的。
