Almadori Erika, Mastroberardino Serena, Botta Fabiano, Brunetti Riccardo, Lupiáñez Juan, Spence Charles, Santangelo Valerio
Neuroimaging Laboratory, IRCCS Santa Lucia Foundation, Via Ardeatina 306, 00179 Rome, Italy.
Department of Psychology, School of Medicine & Psychology, Sapienza University of Rome, Via dei Marsi 78, 00185 Rome, Italy.
Brain Sci. 2021 Sep 13;11(9):1206. doi: 10.3390/brainsci11091206.
Object sounds can enhance the attentional selection and perceptual processing of semantically-related visual stimuli. However, it is currently unknown whether crossmodal semantic congruence also affects the post-perceptual stages of information processing, such as short-term memory (STM), and whether this effect is modulated by the object consistency with the background visual scene. In two experiments, participants viewed everyday visual scenes for 500 ms while listening to an object sound, which could either be semantically related to the object that served as the STM target at retrieval or not. This defined crossmodal semantically cued vs. uncued targets. The target was either in- or out-of-context with respect to the background visual scene. After a maintenance period of 2000 ms, the target was presented in isolation against a neutral background, in either the same or different spatial position as in the original scene. The participants judged the same vs. different position of the object and then provided a confidence judgment concerning the certainty of their response. The results revealed greater accuracy when judging the spatial position of targets paired with a semantically congruent object sound at encoding. This crossmodal facilitatory effect was modulated by whether the target object was in- or out-of-context with respect to the background scene, with out-of-context targets reducing the facilitatory effect of object sounds. Overall, these findings suggest that the presence of the object sound at encoding facilitated the selection and processing of the semantically related visual stimuli, but this effect depends on the semantic configuration of the visual scene.
物体声音能够增强对语义相关视觉刺激的注意选择和知觉加工。然而，目前尚不清楚跨通道语义一致性是否也会影响信息加工的知觉后阶段，如短期记忆（STM），以及这种效应是否会受到物体与背景视觉场景一致性的调节。在两项实验中，参与者在观看日常视觉场景500毫秒的同时聆听物体声音，该声音在检索时可能与作为STM目标的物体语义相关，也可能无关。这定义了跨通道语义提示与未提示的目标。目标相对于背景视觉场景要么符合情境，要么不符合情境。在2000毫秒的保持期后，目标单独呈现在中性背景上，其空间位置与原始场景相同或不同。参与者判断物体的位置相同还是不同，然后对其反应的确定性提供置信度判断。结果显示，在编码时判断与语义一致的物体声音配对的目标的空间位置时，准确性更高。这种跨通道促进效应受到目标物体相对于背景场景是否符合情境的调节，不符合情境的目标会降低物体声音的促进效应。总体而言，这些发现表明，编码时物体声音的存在促进了语义相关视觉刺激的选择和加工，但这种效应取决于视觉场景的语义配置。
