Psychological and Brain Sciences, George Washington University, Washington, DC, USA.
School of Psychology, University of East Anglia, Norwich, UK.
Atten Percept Psychophys. 2022 May;84(4):1317-1327. doi: 10.3758/s13414-022-02488-1. Epub 2022 Apr 21.
Semantic information about objects, events, and scenes influences how humans perceive, interact with, and navigate the world. The semantic information about any object or event can be highly complex and frequently draws on multiple sensory modalities, which makes it difficult to quantify. Past studies have primarily relied on either a simplified binary classification of semantic relatedness based on category or on algorithmic values based on text corpora rather than human perceptual experience and judgement. With the aim to further accelerate research into multisensory semantics, we created a constrained audiovisual stimulus set and derived similarity ratings between items within three categories (animals, instruments, household items). A set of 140 participants provided similarity judgments between sounds and images. Participants either heard a sound (e.g., a meow) and judged which of two pictures of objects (e.g., a picture of a dog and a duck) it was more similar to, or saw a picture (e.g., a picture of a duck) and selected which of two sounds it was more similar to (e.g., a bark or a meow). Judgements were then used to calculate similarity values of any given cross-modal pair. An additional 140 participants provided word judgement to calculate similarity of word-word pairs. The derived and reported similarity judgements reflect a range of semantic similarities across three categories and items, and highlight similarities and differences among similarity judgments between modalities. We make the derived similarity values available in a database format to the research community to be used as a measure of semantic relatedness in cognitive psychology experiments, enabling more robust studies of semantics in audiovisual environments.
人类对物体、事件和场景的语义信息的感知、交互和导航方式会受到影响。任何物体或事件的语义信息都可能非常复杂，并且经常涉及多种感觉模式，这使得其难以量化。过去的研究主要依赖于基于类别或基于文本语料库的算法值的简化二元分类，而不是基于人类的感知经验和判断。为了进一步加速多感官语义的研究，我们创建了一个受限制的视听刺激集，并在三个类别（动物、乐器、家用物品）内得出了项目之间的相似性评分。一组 140 名参与者对声音和图像之间进行了相似性判断。参与者要么听到声音（例如，猫叫声），并判断两个物体图片（例如，狗和鸭子的图片）中哪个与声音更相似，要么看到图片（例如，鸭子的图片），并选择哪个声音与图片更相似（例如，狗叫声或猫叫声）。然后，根据判断结果计算任何给定的跨模态对的相似性值。另外 140 名参与者提供单词判断，以计算单词-单词对的相似性。得出和报告的相似性判断反映了三个类别和项目之间的一系列语义相似性，并突出了不同模态之间的相似性和差异。我们将得出的相似性值以数据库的形式提供给研究社区，作为认知心理学实验中语义相关性的度量，从而能够在视听环境中更深入地研究语义。
