Systems Neuroscience and Neurotechnology Unit, Faculty of Medicine, Saarland University & School of Engineering, htw saar, Homburg/Saar, Germany.
Center for Digital Neurotechnologies Saar, Homburg/Saar & Saarbruecken, Germany.
J Neural Eng. 2024 Oct 29;21(5). doi: 10.1088/1741-2552/ad83f5.
To create highly immersive experiences in virtual reality (VR) it is important to not only include the visual sense but also to involve multimodal sensory input. To achieve optimal results, the temporal and spatial synchronization of these multimodal inputs is critical. It is therefore necessary to find methods to objectively evaluate the synchronization of VR experiences with a continuous tracking of the user.In this study a passive touch experience was incorporated in a visual-tactile VR setup using VR glasses and tactile sensations in mid-air. Inconsistencies of multimodal perception were intentionally integrated into a discrimination task. The participants' electroencephalogram (EEG) was recorded to obtain neural correlates of visual-tactile mismatch situations.The results showed significant differences in the event-related potentials (ERP) between match and mismatch situations. A biphasic ERP configuration consisting of a positivity at 120 ms and a later negativity at 370 ms was observed following a visual-tactile mismatch.This late negativity could be related to the N400 that is associated with semantic incongruency. These results provide a promising approach towards the objective evaluation of visual-tactile synchronization in virtual experiences.
为了在虚拟现实（VR）中创造高度身临其境的体验，不仅要包括视觉感知，还要涉及多模态感官输入。为了达到最佳效果，这些多模态输入的时间和空间同步至关重要。因此，有必要找到方法来客观评估 VR 体验的同步，实现对用户的连续跟踪。
在这项研究中，使用 VR 眼镜和空中触觉，在视觉-触觉 VR 设置中纳入了被动触摸体验。在辨别任务中，故意将多模态感知的不一致性集成在一起。记录参与者的脑电图（EEG）以获取视觉-触觉不匹配情况下的神经相关性。
结果表明，在匹配和不匹配情况下，事件相关电位（ERP）存在显著差异。在视觉-触觉不匹配后观察到双相 ERP 配置，包括 120ms 的正性和 370ms 的后期负性。
这种后期负性可能与 N400 有关，N400 与语义不一致有关。这些结果为客观评估虚拟体验中的视觉-触觉同步提供了一种有前途的方法。
