Ermert Cosima A, Yadav Manuj, Ehret Jonathan, Mohanathasan Chinthusa, Bönsch Andrea, Kuhlen Torsten W, Schlittmeier Sabine J, Fels Janina
Institute for Hearing Technology and Acoustics, RWTH Aachen University, Aachen, Germany.
Visual Computing Institute, RWTH Aachen University, Aachen, Germany.
PLoS One. 2025 Aug 22;20(8):e0330693. doi: 10.1371/journal.pone.0330693. eCollection 2025.
Virtual reality (VR) environments are frequently used in auditory and cognitive research to imitate real-life scenarios. The visual component in VR has the potential to affect how auditory information is processed, especially if incongruences between the visual and auditory information occur. This study investigated how audiovisual incongruence in VR implemented with a head-mounted display (HMD) affects verbal short-term memory compared to presentation of the same material over traditional computer monitors. Two experiments were conducted with both these display devices and two types of audiovisual incongruences: angle (Exp 1) and voice (Exp 2) incongruence. To quantify short-term memory, an audiovisual verbal serial recall (avVSR) task was developed where an embodied conversational agent (ECA) was animated to speak a digit sequence, which participants had to remember. The results showed no effect of the display devices on the proportion of correctly recalled digits overall, although subjective evaluations showed a higher sense of presence in the HMD condition. For the extreme conditions of angle incongruence in the computer monitor presentation, the proportion of correctly recalled digits increased marginally, presumably due to raised attention, but the effect size was negligible. Response times were not affected by incongruences in either display device across both experiments. These findings suggest that at least for the conditions studied here, the avVSR task is robust against angle and voice audiovisual incongruences in both HMD and computer monitor displays.
虚拟现实（VR）环境常用于听觉和认知研究，以模拟现实生活场景。VR中的视觉成分有可能影响听觉信息的处理方式，特别是当视觉和听觉信息之间出现不一致时。本研究调查了与通过传统电脑显示器呈现相同材料相比，使用头戴式显示器（HMD）实现的VR中的视听不一致如何影响言语短期记忆。使用这两种显示设备进行了两项实验，并设置了两种类型的视听不一致：角度（实验1）和声音（实验2）不一致。为了量化短期记忆，开发了一种视听言语序列回忆（avVSR）任务，其中一个具身对话代理（ECA）被设置为说出一个数字序列，参与者必须记住这个序列。结果显示，显示设备对总体正确回忆数字的比例没有影响，尽管主观评价显示在HMD条件下有更高的临场感。对于电脑显示器呈现中角度不一致的极端情况，正确回忆数字的比例略有增加，可能是由于注意力提高，但效应大小可以忽略不计。在两个实验中，响应时间均不受任何一种显示设备中不一致情况的影响。这些发现表明，至少对于此处研究的条件，avVSR任务在HMD和电脑显示器显示中对角度和声音视听不一致具有稳健性。
