Van Audenhaege Alice, Mattioni Stefania, Cerpelloni Filippo, Gau Remi, Szmalec Arnaud, Collignon Olivier
Institute of Psychology (IPSY) & Institute of Neuroscience (IoNS), UCLouvain, Louvain-la-Neuve 1348, Belgium.
Department of Experimental Psychology, Ghent University, Ghent 9000, Belgium.
J Neurosci. 2025 Apr 30. doi: 10.1523/JNEUROSCI.1415-24.2025.
Speech is a multisensory signal that can be extracted from the voice and the lips. Previous studies suggested that occipital and temporal regions encode both auditory and visual speech features but their location and nature remain unclear. We characterized brain activity using fMRI (in male and female) to functionally and individually define bilateral Fusiform Face Areas (FFA), the left word-selective ventral occipito-temporal cortex (word-VOTC), an audio-visual speech region in the left Superior Temporal Sulcus (lSTS) and control regions in bilateral scene-selective Para-hippocampal Place Areas (PPA). In these regions, we performed multivariate patterns classification of corresponding phonemes (speech sounds) and visemes (lip movements). We observed that the word-VOTC and lSTS represent phonological information from both vision and sounds. The multisensory nature of phonological representations appeared selective to the word-VOTC, as we found viseme but not phoneme representation in adjacent FFA, while PPA did not encode phonology in any modality. Interestingly, cross-modal decoding revealed aligned phonological representations across the senses in lSTS, but not in word-VOTC. A whole-brain cross-modal searchlight analysis additionally revealed aligned audio-visual phonological representations in bilateral pSTS and left somato-motor cortex overlapping with oro-facial articulators. Altogether, our results demonstrate that auditory and visual phonology are represented in the word-VOTC, extending its functional coding beyond orthography. The geometries of auditory and visual representations do not align in the word-VOTC as they do in the STS and left somato-motor cortex, suggesting distinct multisensory representations across a distributed phonological network. Speech is a multisensory signal that can be extracted from the voice and the lips. Which brain regions encode both visual and auditory speech representations? We show that the written word-selective ventral occipito-temporal (word-VOTC) and the left Superior Temporal Sulcus (lSTS) both process phonological information from speech sounds and lip movements. However, while the lSTS aligns these representations across the senses, the word-VOTC does not, indicating different encoding mechanisms. These findings extend the functional role of the word-VOTC beyond reading. An additional whole-brain approach reveals shared representations in bilateral superior temporal cortex and left somato-motor cortex, indicating a distributed network for multisensory phonology.
言语是一种多感官信号，可从语音和嘴唇动作中提取。先前的研究表明，枕叶和颞叶区域对听觉和视觉言语特征都进行编码，但其位置和性质仍不清楚。我们使用功能磁共振成像（fMRI，涵盖男性和女性）来表征大脑活动，以便从功能上和个体角度定义双侧梭状面孔区（FFA）、左侧单词选择性腹侧枕颞叶皮层（单词-VOTC）、左侧颞上沟中的一个视听言语区域（lSTS）以及双侧场景选择性海马旁回区域（PPA）中的控制区域。在这些区域，我们对相应的音素（语音）和视素（嘴唇动作）进行多变量模式分类。我们观察到，单词-VOTC和lSTS代表来自视觉和声音的语音信息。语音表征的多感官性质似乎对单词-VOTC具有选择性，因为我们在相邻的FFA中发现了视素表征而非音素表征，而PPA在任何模态下都不编码语音。有趣的是，跨模态解码揭示了lSTS中跨感官的对齐语音表征，但在单词-VOTC中未发现。全脑跨模态探照灯分析还揭示了双侧颞上沟后部（pSTS）和左侧躯体运动皮层中与口面部发音器重叠的对齐视听语音表征。总体而言，我们的结果表明，听觉和视觉语音在单词-VOTC中得到表征，将其功能编码扩展到了正字法之外。听觉和视觉表征的几何结构在单词-VOTC中不像在颞上沟和左侧躯体运动皮层中那样对齐，这表明在一个分布式语音网络中存在不同的多感官表征。言语是一种可从语音和嘴唇动作中提取的多感官信号。哪些脑区对视觉和听觉言语表征都进行编码？我们表明，书写单词选择性腹侧枕颞叶（单词-VOTC）和左侧颞上沟（lSTS）都处理来自语音和嘴唇动作的语音信息。然而，虽然lSTS在不同感官间对齐这些表征，但单词-VOTC并非如此，这表明存在不同的编码机制。这些发现将单词-VOTC的功能作用扩展到了阅读之外。另一种全脑方法揭示了双侧颞叶上部皮层和左侧躯体运动皮层中的共享表征，表明存在一个用于多感官语音的分布式网络。
