Basque Center on Cognition, Brain and Language, 20009 Donostia-San Sebastián, Spain.
Departamento de Lengua Vasca y Comunicación, University of the Basque Country (UPV/EHU), Leioa, 48940 Spain.
Proc Natl Acad Sci U S A. 2022 Sep 6;119(36):e2203906119. doi: 10.1073/pnas.2203906119. Epub 2022 Aug 29.
We exploit the phenomenon of cross-modal, cross-language activation to examine the dynamics of language processing. Previous within-language work showed that seeing a sign coactivates phonologically related signs, just as hearing a spoken word coactivates phonologically related words. In this study, we conducted a series of eye-tracking experiments using the visual world paradigm to investigate the time course of cross-language coactivation in hearing bimodal bilinguals (Spanish-Spanish Sign Language) and unimodal bilinguals (Spanish/Basque). The aim was to gauge whether (and how) seeing a sign could coactivate words and, conversely, how hearing a word could coactivate signs and how such cross-language coactivation patterns differ from within-language coactivation. The results revealed cross-language, cross-modal activation in both directions. Furthermore, comparison with previous findings of within-language lexical coactivation for spoken and signed language showed how the impact of temporal structure changes in different modalities. Spoken word activation follows the temporal structure of that word only when the word itself is heard; for signs, the temporal structure of the sign does not govern the time course of lexical access (location coactivation precedes handshape coactivation)-even when the sign is seen. We provide evidence that, instead, this pattern of activation is motivated by how common in the lexicon the sublexical units of the signs are. These results reveal the interaction between the perceptual properties of the explicit signal and structural linguistic properties. Examining languages across modalities illustrates how this interaction impacts language processing.
我们利用跨模态、跨语言激活的现象来研究语言处理的动态。先前的语言内研究表明，看到一个符号会激活语音相关的符号，就像听到一个口头单词会激活语音相关的单词一样。在这项研究中，我们使用视觉世界范式进行了一系列眼动追踪实验，以调查听觉双语者（西班牙语-西班牙语手语）和单语双语者（西班牙语/巴斯克语）的跨语言激活的时间进程。目的是衡量看到一个符号是否能激活单词，反之，听到一个单词是否能激活符号，以及这种跨语言的激活模式与语言内激活有何不同。结果显示了两种方向的跨语言、跨模态激活。此外，与之前关于口语和手语的语言内词汇激活的发现进行比较，揭示了不同模态中时间结构变化的影响。只有当单词本身被听到时，口语单词的激活才遵循该单词的时间结构；对于手语来说，手语的时间结构并不支配词汇访问的时间进程（位置激活先于手形激活）——即使看到手语也是如此。我们提供的证据表明，这种激活模式是由手语的亚词汇单元在词汇中的常见程度决定的。这些结果揭示了显式信号的感知属性和结构语言学属性之间的相互作用。跨模态研究语言表明了这种相互作用如何影响语言处理。
