Neurocomputational and Intelligent Signal Processing Research Group (NECOTIS), Département de Génie Électrique et de Génie Informatique, Faculté de Génie, Université de Sherbrooke Sherbrooke, QC, Canada.
Neurocomputational and Intelligent Signal Processing Research Group (NECOTIS), Département de Génie Électrique et de Génie Informatique, Faculté de Génie, Université de Sherbrooke Sherbrooke, QC, Canada ; Neuroscience Lab, Département de Sciences Biologiques, Université de Montréal Montréal, QC, Canada.
Front Hum Neurosci. 2014 May 30;8:352. doi: 10.3389/fnhum.2014.00352. eCollection 2014.
This article investigates the cross-modal correspondences between musical timbre and shapes. Previously, such features as pitch, loudness, light intensity, visual size, and color characteristics have mostly been used in studies of audio-visual correspondences. Moreover, in most studies, simple stimuli e.g., simple tones have been utilized. In this experiment, 23 musical sounds varying in fundamental frequency and timbre but fixed in loudness were used. Each sound was presented once against colored shapes and once against grayscale shapes. Subjects had to select the visual equivalent of a given sound i.e., its shape, color (or grayscale) and vertical position. This scenario permitted studying the associations between normalized timbre and visual shapes as well as some of the previous findings for more complex stimuli. One hundred and nineteen subjects (31 females and 88 males) participated in the online experiment. Subjects included 36 claimed professional musicians, 47 claimed amateur musicians, and 36 claimed non-musicians. Thirty-one subjects have also claimed to have synesthesia-like experiences. A strong association between timbre of envelope normalized sounds and visual shapes was observed. Subjects have strongly associated soft timbres with blue, green or light gray rounded shapes, harsh timbres with red, yellow or dark gray sharp angular shapes and timbres having elements of softness and harshness together with a mixture of the two previous shapes. Color or grayscale had no effect on timbre-shape associations. Fundamental frequency was not associated with height, grayscale or color. The significant correspondence between timbre and shape revealed by the present work allows designing substitution systems which might help the blind to perceive shapes through timbre.
本文研究了音乐音色与形状之间的跨模态对应关系。以前，在视听对应关系的研究中，主要使用了音高、响度、光强度、视觉大小和颜色特征等特征。此外，在大多数研究中，使用了简单的刺激，例如简单的音调。在这个实验中，使用了 23 种基频和音色不同但响度固定的音乐声音。每个声音都以彩色形状和灰度形状各呈现一次。被试必须选择给定声音的视觉等效物，即其形状、颜色（或灰度）和垂直位置。这种情况允许研究归一化音色与视觉形状之间的关联，以及之前对更复杂刺激的一些发现。119 名受试者（31 名女性和 88 名男性）参加了在线实验。受试者包括 36 名自称专业音乐家、47 名自称业余音乐家和 36 名自称非音乐家。31 名受试者还声称有类联觉体验。观察到归一化声音包络音色与视觉形状之间存在强烈关联。被试强烈地将柔和的音色与蓝色、绿色或浅灰色的圆形形状联系起来，将刺耳的音色与红色、黄色或深灰色的尖锐角形状联系起来，将具有柔和和刺耳元素的音色与两种先前形状的混合联系起来。颜色或灰度对音色-形状关联没有影响。基频与高度、灰度或颜色没有关联。本工作揭示的音色与形状之间的显著对应关系，允许设计替代系统，帮助盲人通过音色感知形状。
