Human Technology Center, RWTH Aachen University, Theaterplatz 14, 52056 Aachen, Germany; Department of Psychiatry, Psychotherapy and Psychosomatics, School of Medicine, RWTH Aachen University, Pauwelsstraße 30, 52074 Aachen, Germany; Department of Systems Neuroscience, University Medical Center Hamburg-Eppendorf, 20246 Hamburg, Germany.
School of Medicine, RWTH Aachen University, Pauwelsstraße 30, 52074 Aachen, Germany.
Brain Lang. 2020 Jun;205:104772. doi: 10.1016/j.bandl.2020.104772. Epub 2020 Feb 29.
This paper presents an fMRI study on healthy adult understanding of metaphors in multimodal communication. We investigated metaphors expressed either only in coverbal gestures ("monomodal metaphors") or in speech with accompanying gestures ("multimodal metaphors"). Monomodal metaphoric gestures convey metaphoric information not expressed in the accompanying speech (e.g. saying the non-metaphoric utterance, "She felt bad" while dropping down the hand with palm facing up; here, the gesture alone indicates metaphoricity), whereas coverbal gestures in multimodal metaphors indicate metaphoricity redundant to the speech (e.g. saying the metaphoric utterance, "Her spirits fell" while dropping the hand with palm facing up). In other words, in monomodal metaphors, gestures add information not spoken, whereas the gestures in multimodal metaphors can be redundant to the spoken content. Understanding and integrating the information in each modality, here spoken and visual, is important in multimodal communication, but most prior studies have only considered multimodal metaphors where the gesture is redundant to what is spoken. Our participants watched audiovisual clips of an actor speaking while gesturing. We found that abstract metaphor comprehension recruited the lateral superior/middle temporal cortices, regardless of the modality in which the conceptual metaphor is expressed. These results suggest that abstract metaphors, regardless of modality, involve resources implicated in general semantic processing and are consistent with the role of these areas in supramodal semantic processing as well as the theory of embodied cognition.
这篇论文介绍了一项针对健康成年人理解多模态交流中隐喻的 fMRI 研究。我们研究了仅通过口头语言表达的隐喻（“单模态隐喻”）和伴随手势的语言表达的隐喻（“多模态隐喻”）。单模态隐喻性手势传达了未在伴随言语中表达的隐喻信息（例如，说出非隐喻性话语“她感觉很糟糕”，同时向上伸出手掌；在这里，仅手势就表示隐喻性），而多模态隐喻中的口头语言手势则表示与言语冗余的隐喻性（例如，说出隐喻性话语“她的精神低落”，同时向上伸出手掌）。换句话说，在单模态隐喻中，手势增加了未说出的信息，而多模态隐喻中的手势则与所说内容冗余。理解和整合每种模态（这里是口语和视觉）的信息在多模态交流中很重要，但大多数先前的研究仅考虑了手势与所说内容冗余的多模态隐喻。我们的参与者观看了演员说话同时手势的视听片段。我们发现，抽象隐喻理解招募了外侧上/中颞叶皮质，而不论概念隐喻以哪种模态表达。这些结果表明，抽象隐喻，无论模态如何，都涉及到一般语义处理中涉及的资源，这与这些区域在超模态语义处理中的作用以及具身认知理论一致。
