Upadhyayula Aditya, Cohn Neil
Department of Psychological & Brain Sciences, Washington University in St. Louis.
Department of Communication & Cognition, Tilburg University.
Cogn Sci. 2025 May;49(5):e70050. doi: 10.1111/cogs.70050.
Theories of visual narrative comprehension have advocated for a hierarchical grammar-based comprehension mechanism, but only limited work has investigated this hierarchy. Here, we provide a computational framework inspired by computational psycholinguistics to address hierarchy in visual narratives. The predictions generated by this framework were compared against behavior data to draw inferences about the hierarchical properties of visual narratives. A segmentation task-where participants ranked all possible segmental boundaries-demonstrated that participants' preferences were predicted by visual narrative grammar. Three kinds of models using surprisal theory-an Earley parser, a hidden Markov model (HMM), and an n-gram model-were then used to generate segmentation preferences for the same task. Earley parser's preferences were based on a hierarchical grammar with recursion properties, while the HMM and the n-grams used a flattened grammar for visual narrative comprehension. Given the differences in the mechanics of these models, contrasting their predictions against behavior data could provide crucial insights into understanding the underlying mechanisms of visual narrative comprehension. By investigating grammatical systems outside of language, this research provides new directions to explore the generic makeup of the cognitive structure of mental representations.
视觉叙事理解理论主张基于层次语法的理解机制，但仅有有限的研究探讨了这种层次结构。在此，我们提供一个受计算心理语言学启发的计算框架，以解决视觉叙事中的层次结构问题。将该框架生成的预测与行为数据进行比较，以推断视觉叙事的层次属性。一项分割任务（参与者对所有可能的分割边界进行排序）表明，参与者的偏好可由视觉叙事语法预测。然后使用三种基于惊奇理论的模型——一个厄立 parser、一个隐马尔可夫模型（HMM）和一个n-gram模型——为同一任务生成分割偏好。厄立 parser的偏好基于具有递归属性的层次语法，而HMM和n-gram模型则使用扁平语法进行视觉叙事理解。鉴于这些模型机制的差异，将它们的预测与行为数据进行对比，可为理解视觉叙事理解的潜在机制提供关键见解。通过研究语言之外的语法系统，本研究为探索心理表征认知结构的一般构成提供了新方向。
