Department of Psychology, Tufts University, Medford, MA 02155, USA.
Cogn Psychol. 2012 Aug;65(1):1-38. doi: 10.1016/j.cogpsych.2012.01.003. Epub 2012 Mar 2.
Just as syntax differentiates coherent sentences from scrambled word strings, the comprehension of sequential images must also use a cognitive system to distinguish coherent narrative sequences from random strings of images. We conducted experiments analogous to two classic studies of language processing to examine the contributions of narrative structure and semantic relatedness to processing sequential images. We compared four types of comic strips: (1) Normal sequences with both structure and meaning, (2) Semantic Only sequences (in which the panels were related to a common semantic theme, but had no narrative structure), (3) Structural Only sequences (narrative structure but no semantic relatedness), and (4) Scrambled sequences of randomly-ordered panels. In Experiment 1, participants monitored for target panels in sequences presented panel-by-panel. Reaction times were slowest to panels in Scrambled sequences, intermediate in both Structural Only and Semantic Only sequences, and fastest in Normal sequences. This suggests that both semantic relatedness and narrative structure offer advantages to processing. Experiment 2 measured ERPs to all panels across the whole sequence. The N300/N400 was largest to panels in both the Scrambled and Structural Only sequences, intermediate in Semantic Only sequences and smallest in the Normal sequences. This implies that a combination of narrative structure and semantic relatedness can facilitate semantic processing of upcoming panels (as reflected by the N300/N400). Also, panels in the Scrambled sequences evoked a larger left-lateralized anterior negativity than panels in the Structural Only sequences. This localized effect was distinct from the N300/N400, and appeared despite the fact that these two sequence types were matched on local semantic relatedness between individual panels. These findings suggest that sequential image comprehension uses a narrative structure that may be independent of semantic relatedness. Altogether, we argue that the comprehension of visual narrative is guided by an interaction between structure and meaning.
就像语法将连贯的句子与乱序的单词串区分开来一样，对连续图像的理解也必须使用认知系统将连贯的叙事序列与随机的图像序列区分开来。我们进行了类似于语言处理的两项经典研究的实验，以检验叙事结构和语义相关性对处理连续图像的贡献。我们比较了四种漫画类型：（1）具有结构和意义的正常序列，（2）语义仅序列（其中面板与共同的语义主题相关，但没有叙事结构），（3）结构仅序列（叙事结构但没有语义相关性），以及（4）随机排序的面板乱序序列。在实验 1 中，参与者在逐面板呈现的序列中监视目标面板。在乱序序列中，面板的反应时间最慢，在结构仅序列和语义仅序列中反应时间居中，在正常序列中反应时间最快。这表明语义相关性和叙事结构都为处理提供了优势。实验 2 测量了整个序列中所有面板的 ERP。在乱序和结构仅序列中，N300/N400 对面板的影响最大，在语义仅序列中居中，在正常序列中最小。这意味着叙事结构和语义相关性的结合可以促进对后续面板的语义处理（如 N300/N400 所反映的那样）。此外，乱序序列中的面板比结构仅序列中的面板引起更大的左侧前负性。这种局部效应与 N300/N400 不同，尽管这两种序列类型在面板之间的局部语义相关性上是匹配的。这些发现表明，连续图像理解使用的叙事结构可能独立于语义相关性。总的来说，我们认为视觉叙事的理解是由结构和意义之间的相互作用指导的。
