IEEE Trans Vis Comput Graph. 2022 May;28(5):2080-2090. doi: 10.1109/TVCG.2022.3150514. Epub 2022 Apr 8.
Studies in virtual reality (VR) have introduced numerous multisensory simulation techniques for more immersive VR experiences. However, although they primarily focus on expanding sensory types or increasing individual sensory quality, they lack consensus in designing appropriate interactions between different sensory stimuli. This paper explores how the congruence between auditory and visual (AV) stimuli, which are the sensory stimuli typically provided by VR devices, affects the cognition and experience of VR users as a critical interaction factor in promoting multisensory integration. We defined the types of (in)congruence between AV stimuli, and then designed 12 virtual spaces with different types or degrees of congruence between AV stimuli. We then evaluated the presence, immersion, motion sickness, and cognition changes in each space. We observed the following key findings: 1) there is a limit to the degree of temporal or spatial incongruence that can be tolerated, with few negative effects on user experience until that point is exceeded; 2) users are tolerant of semantic incongruence; 3) a simulation that considers synesthetic congruence contributes to the user's sense of immersion and presence. Based on these insights, we identified the essential considerations for designing sensory simulations in VR and proposed future research directions.
虚拟现实（VR）中的研究已经引入了许多多感官模拟技术，以提供更具沉浸感的 VR 体验。然而，尽管它们主要侧重于扩展感官类型或提高单个感官质量，但在设计不同感官刺激之间的适当交互方面缺乏共识。本文探讨了听觉和视觉（AV）刺激之间的一致性，即 VR 设备通常提供的感官刺激，如何作为促进多感官整合的关键交互因素，影响 VR 用户的认知和体验。我们定义了 AV 刺激之间的（不）一致性类型，然后设计了 12 个具有不同类型或 AV 刺激一致性程度的虚拟空间。然后，我们评估了每个空间中的存在、沉浸感、晕动病和认知变化。我们观察到以下关键发现：1）可以容忍的时间或空间不一致程度有限，在超过该点之前，对用户体验的负面影响很少；2）用户容忍语义不一致性；3）考虑到联觉一致性的模拟有助于用户的沉浸感和存在感。基于这些见解，我们确定了在 VR 中设计感官模拟的基本考虑因素，并提出了未来的研究方向。
