Lembke Sven-Amin
Cambridge School of Creative Industries, Anglia Ruskin University, Cambridge, United Kingdom.
Front Psychol. 2025 Feb 7;16:1520209. doi: 10.3389/fpsyg.2025.1520209. eCollection 2025.
Communication through auditory cues often evokes associations to other sensory modalities. In film music, for instance, a descending pitch contour commonly resembles a falling motion. Such crossmodal associations to physical actions or shapes are here termed and can naturally occur in environmental sounds. Little is known about how reliably listeners perceive gestures in such environmental contexts and how salient the gesture-relevant auditory feature needs to be. This article reports on an exploratory study concerning the identification of sound gestures by crossmodal matching using analogous visualizations. The study considered gesture-related factors, such as auditory salience and contour complexity, and explored whether a concurrent evaluation of features related to the environmental sound source or cause would affect gesture identification. Twenty untrained listeners evaluated sound gestures occurring in environmental sounds, e.g., pitch contour when switching a vacuum cleaner on and off, loudness contour of a ball dropping. Participants evaluated 28 environmental sounds in three variants (original, isolated gesture, hybrid) and had to identify the sound gesture among four visualized options while also inferring the underlying environmental source or cause through verbal description and rating their confidence in identifying the source/cause. Based on features describing the macro contour of gestures, participants correctly identified 81-83% of all gestures. Manipulated sounds that emphasized gesture salience yielded only slight improvements of identification accuracy compared to original environmental sounds. Participants were more confident in identifying the source/cause in sounds containing pitch gestures than those containing loudness gestures, while lexical and semantic diversity in describing underlying materials (source) and actions (cause) varied considerably. For both groups, however, measures for gesture identification and the evaluation of underlying materials and actions correlated only weakly, suggesting task independence. Overall, findings suggest that untrained listeners perceive sound gestures in environmental sounds and can reliably use them to form crossmodal associations, while also evaluating properties related to the sound source and cause. For one, the perception of environmental sounds may evoke crossmodal links, while the reliable identification of sound gestures highlights their utility to crossmodal control or search interfaces.
通过听觉线索进行的交流常常会引发与其他感官模态的联想。例如，在电影音乐中，音高轮廓下降通常类似于下落动作。这种与身体动作或形状的跨模态联想在此被称为 ，并且自然地会出现在环境声音中。对于听众在这种环境背景下感知手势的可靠性以及与手势相关的听觉特征需要多显著，我们知之甚少。本文报告了一项探索性研究，该研究通过使用类似的可视化方法进行跨模态匹配来识别声音手势。该研究考虑了与手势相关的因素，如听觉显著性和轮廓复杂性，并探讨了对与环境声源或原因相关的特征进行并发评估是否会影响手势识别。20名未经训练的听众评估了环境声音中出现的声音手势，例如打开和关闭吸尘器时的音高轮廓、球掉落时的响度轮廓。参与者评估了28种环境声音的三种变体（原始、孤立手势、混合），并必须在四个可视化选项中识别声音手势，同时通过口头描述推断潜在的环境声源或原因，并对识别声源/原因的信心进行评级。基于描述手势宏观轮廓的特征，参与者正确识别了所有手势的81 - 83%。与原始环境声音相比，强调手势显著性的处理声音仅使识别准确率略有提高。参与者在识别包含音高手势的声音中的声源/原因时比识别包含响度手势的声音时更有信心，而在描述潜在材料（声源）和动作（原因）时的词汇和语义多样性差异很大。然而，对于两组来说，手势识别措施与对潜在材料和动作的评估之间的相关性都很弱，表明任务具有独立性。总体而言，研究结果表明，未经训练的听众能够感知环境声音中的声音手势，并能可靠地利用它们形成跨模态联想，同时还能评估与声源和原因相关的属性。一方面，对环境声音的感知可能会引发跨模态联系，而声音手势的可靠识别突出了它们在跨模态控制或搜索界面中的效用。
