Kelly Spencer, Healey Meghan, Özyürek Asli, Holler Judith
Department of Psychology, Neuroscience Program, Colgate University, 105D Olin Hall, Hamilton, NY, USA,
Psychon Bull Rev. 2015 Apr;22(2):517-23. doi: 10.3758/s13423-014-0681-7.
Hand gestures and speech form a single integrated system of meaning during language comprehension, but is gesture processed with speech in a unique fashion? We had subjects watch multimodal videos that presented auditory (words) and visual (gestures and actions on objects) information. Half of the subjects related the audio information to a written prime presented before the video, and the other half related the visual information to the written prime. For half of the multimodal video stimuli, the audio and visual information contents were congruent, and for the other half, they were incongruent. For all subjects, stimuli in which the gestures and actions were incongruent with the speech produced more errors and longer response times than did stimuli that were congruent, but this effect was less prominent for speech-action stimuli than for speech-gesture stimuli. However, subjects focusing on visual targets were more accurate when processing actions than gestures. These results suggest that although actions may be easier to process than gestures, gestures may be more tightly tied to the processing of accompanying speech.
在语言理解过程中，手势和言语构成了一个单一的综合意义系统，但是手势是以一种独特的方式与言语一起被处理的吗？我们让受试者观看多模态视频，这些视频呈现听觉（单词）和视觉（手势以及对物体的动作）信息。一半受试者将音频信息与视频之前呈现的书面启动刺激相关联，另一半受试者将视觉信息与书面启动刺激相关联。对于一半的多模态视频刺激，音频和视觉信息内容是一致的，而对于另一半，它们是不一致的。对于所有受试者来说，手势和动作与言语不一致的刺激比一致的刺激产生更多错误和更长的反应时间，但这种效应在言语 - 动作刺激中比在言语 - 手势刺激中不那么明显。然而，专注于视觉目标的受试者在处理动作时比处理手势时更准确。这些结果表明，虽然动作可能比手势更容易处理，但手势可能与伴随言语的处理联系更紧密。
