Cohn Neil, Kutas Marta
Center for Research in Language, University of California, San Diego, La Jolla, CA USA.
Tilburg Center for Cognition and Communication, Tilburg University, P.O. Box 90153, 5000 LE Tilburg, the Netherlands.
Cogn Res Princ Implic. 2017;2(1):27. doi: 10.1186/s41235-017-0064-5. Epub 2017 May 24.
Visual narratives sometimes depict successive images with different characters in the same physical space; corpus analysis has revealed that this occurs more often in Japanese manga than American comics. We used event-related brain potentials to determine whether comprehension of "visual narrative conjunctions" invokes not only incremental mental updating as traditionally assumed, but also, as we propose, "grammatical" combinatoric processing. We thus crossed (non)/conjunction sequences with character (in)/congruity. Conjunctions elicited a larger anterior negativity (300-500 ms) than nonconjunctions, regardless of congruity, implicating "grammatical" processes. Conjunction and incongruity both elicited larger P600s (500-700 ms), indexing updating. Both conjunction effects were modulated by participants' frequency of reading manga while growing up. Greater anterior negativity in frequent manga readers suggests more reliance on combinatoric processing; larger P600 effects in infrequent manga readers suggest more resources devoted to mental updating. As in language comprehension, it seems that processing conjunctions in visual narratives is not just mental updating but also partly grammatical, conditioned by comic readers' experience with specific visual narrative structures.
视觉叙事有时会在同一物理空间中描绘具有不同角色的连续图像;语料库分析表明,这种情况在日本漫画中比在美国漫画中更常见。我们使用事件相关脑电位来确定对“视觉叙事连词”的理解是否不仅如传统假设的那样引发渐进的心理更新,而且如我们所提出的那样还引发“语法”组合处理。因此,我们将(非)连词序列与角色(不)一致性进行了交叉。无论一致性如何,连词比非连词引发更大的前部负波(300 - 500毫秒),这意味着存在“语法”过程。连词和不一致性都引发了更大的P600(500 - 700毫秒),表明在进行更新。两种连词效应都受到参与者成长过程中阅读漫画频率的调节。频繁阅读漫画的读者中更大的前部负波表明更多地依赖组合处理;不频繁阅读漫画的读者中更大的P600效应表明更多资源用于心理更新。与语言理解一样,在视觉叙事中处理连词似乎不仅是心理更新,而且在一定程度上也是语法性的,这取决于漫画读者对特定视觉叙事结构的经验。