Institute for Neural Computation, University of California, San Diego, La Jolla, California.
Department of Electrical and Computer Engineering, University of California, San Diego, La Jolla, California.
Hum Brain Mapp. 2018 May;39(5):2156-2176. doi: 10.1002/hbm.23995. Epub 2018 Feb 6.
Real-world objects approaching or passing by an observer often generate visual, auditory, and tactile signals with different onsets and durations. Prompt detection and avoidance of an impending threat depend on precise binding of looming signals across modalities. Here we constructed a multisensory apparatus to study the spatiotemporal integration of looming visual and tactile stimuli near the face. In a psychophysical experiment, subjects assessed the subjective synchrony between a looming ball and an air puff delivered to the same side of the face with a varying temporal offset. Multisensory stimuli with similar onset times were perceived as completely out of sync and assessed with the lowest subjective synchrony index (SSI). Across subjects, the SSI peaked at an offset between 800 and 1,000 ms, where the multisensory stimuli were perceived as optimally in sync. In an fMRI experiment, tactile, visual, tactile-visual out-of-sync (TVoS), and tactile-visual in-sync (TViS) stimuli were delivered to either side of the face in randomized events. Group-average statistical responses to different stimuli were compared within each surface-based region of interest (sROI) outlined on the cortical surface. Most sROIs showed a preference for contralateral stimuli and higher responses to multisensory than unisensory stimuli. In several bilateral sROIs, particularly the human MT+ complex and V6A, responses to spatially aligned multisensory stimuli (TVoS) were further enhanced when the stimuli were in-sync (TViS), as expressed by TVoS < TViS. This study demonstrates the perceptual and neural mechanisms of multisensory integration near the face, which has potential applications in the development of multisensory entertainment systems and media.
真实世界中的物体接近或经过观察者时，通常会产生具有不同起始和持续时间的视觉、听觉和触觉信号。对即将到来的威胁的快速检测和避免依赖于不同模态中逼近信号的精确绑定。在这里，我们构建了一个多感觉仪器来研究面部附近逼近的视觉和触觉刺激的时空整合。在一项心理物理学实验中，受试者评估了逼近的球和向面部同一侧施加的空气喷射之间的主观同步性，具有不同的时间偏移。具有相似起始时间的多感觉刺激被感知为完全不同步，并具有最低的主观同步指数（SSI）进行评估。在受试者中，SSI 在 800 到 1000 毫秒之间的偏移处达到峰值，在这个偏移处，多感觉刺激被感知为最佳同步。在 fMRI 实验中，触觉、视觉、触觉-视觉失步（TVoS）和触觉-视觉同步（TViS）刺激以随机事件的方式分别传递到面部的两侧。在每个基于表面的感兴趣区（sROI）内比较了对不同刺激的组平均统计响应，这些 sROI 是在皮质表面上勾勒出的。大多数 sROI 表现出对对侧刺激的偏好，并且对多感觉刺激的反应比对单感觉刺激的反应更高。在几个双侧 sROI 中，特别是人类 MT+ 复合体和 V6A，当刺激同步时（TViS），空间上对齐的多感觉刺激（TVoS）的反应进一步增强，如 TVoS<TViS 所示。这项研究展示了面部附近多感觉整合的感知和神经机制，这在多感觉娱乐系统和媒体的发展中具有潜在的应用。
