Lemaitre Guillaume, Scurto Hugo, Françoise Jules, Bevilacqua Frédéric, Houix Olivier, Susini Patrick
Equipe Perception et Design Sonores, STMS-IRCAM-CNRS-UPMC, Institut de Recherche et de Coordination Acoustique Musique, Paris, France.
Equipe Interaction Sons Musique Mouvement, STMS-IRCAM-CNRS-UPMC, Institut de Recherche et de Coordination Acoustique Musique, Paris, France.
PLoS One. 2017 Jul 27;12(7):e0181786. doi: 10.1371/journal.pone.0181786. eCollection 2017.
Communicating an auditory experience with words is a difficult task and, in consequence, people often rely on imitative non-verbal vocalizations and gestures. This work explored the combination of such vocalizations and gestures to communicate auditory sensations and representations elicited by non-vocal everyday sounds. Whereas our previous studies have analyzed vocal imitations, the present research focused on gestural depictions of sounds. To this end, two studies investigated the combination of gestures and non-verbal vocalizations. A first, observational study examined a set of vocal and gestural imitations of recordings of sounds representative of a typical everyday environment (ecological sounds) with manual annotations. A second, experimental study used non-ecological sounds whose parameters had been specifically designed to elicit the behaviors highlighted in the observational study, and used quantitative measures and inferential statistics. The results showed that these depicting gestures are based on systematic analogies between a referent sound, as interpreted by a receiver, and the visual aspects of the gestures: auditory-visual metaphors. The results also suggested a different role for vocalizations and gestures. Whereas the vocalizations reproduce all features of the referent sounds as faithfully as vocally possible, the gestures focus on one salient feature with metaphors based on auditory-visual correspondences. Both studies highlighted two metaphors consistently shared across participants: the spatial metaphor of pitch (mapping different pitches to different positions on the vertical dimension), and the rustling metaphor of random fluctuations (rapidly shaking of hands and fingers). We interpret these metaphors as the result of two kinds of representations elicited by sounds: auditory sensations (pitch and loudness) mapped to spatial position, and causal representations of the sound sources (e.g. rain drops, rustling leaves) pantomimed and embodied by the participants' gestures.
用语言传达听觉体验是一项艰巨的任务，因此，人们常常依赖模仿性的非语言发声和手势。这项研究探索了此类发声和手势的组合，以传达由日常非语音声音引发的听觉感受和表征。我们之前的研究分析了声音模仿，而本研究则聚焦于声音的手势描绘。为此，进行了两项研究来调查手势与非语言发声的组合。第一项观察性研究，对手动标注的一组代表典型日常环境的声音录音（生态声音）的发声和手势模仿进行了研究。第二项实验性研究使用了专门设计参数以引发观察性研究中所强调行为的非生态声音，并采用了定量测量和推断统计。结果表明，这些描绘性手势基于接收者所理解的指涉声音与手势视觉方面之间的系统类比：即听觉 - 视觉隐喻。结果还表明了发声和手势所起的不同作用。发声尽可能忠实地再现指涉声音的所有特征，而手势则基于听觉 - 视觉对应关系，聚焦于一个突出特征并运用隐喻。两项研究都突出了参与者之间一致共享的两种隐喻：音高的空间隐喻（将不同音高映射到垂直维度上的不同位置），以及随机波动的沙沙声隐喻（快速摆动双手和手指）。我们将这些隐喻解释为声音引发的两种表征的结果：映射到空间位置的听觉感受（音高和响度），以及参与者手势所模仿和体现的声源因果表征（如雨滴、沙沙作响的树叶）。
