Setu Jyotirmay Nag, Le Joshua M, Kundu Ripan Kumar, Giesbrecht Barry, Hollerer Tobias, Hoque Khaza Anuarul, Desai Kevin, Quarles John
IEEE Trans Vis Comput Graph. 2025 May;31(5):3014-3024. doi: 10.1109/TVCG.2025.3549850. Epub 2025 Apr 29.
As VR technology advances, the demand for multitasking within virtual environments escalates. Negotiating multiple tasks within the immersive virtual setting presents cognitive challenges, where users experience difficulty executing multiple concurrent tasks. This phenomenon highlights the importance of cognitive functions like attention and working memory, which are vital for navigating intricate virtual environments effectively. In addition to attention and working memory, assessing the extent of physical and mental strain induced by the virtual environment and the concurrent tasks performed by the participant is key. While previous research has focused on investigating factors influencing attention and working memory in virtual reality, more comprehensive approaches addressing the prediction of physical and mental strain alongside these cognitive aspects remain. This gap inspired our investigation, where we utilized an open dataset - VRWalking, which included eye and head tracking and physiological measures like heart rate(HR) and galvanic skin response(GSR). The VRwalking dataset has timestamped labeled data for physical and mental load, working memory, and attention metrics. In our investigation, we employed straightforward deep learning models to predict these labels, achieving noteworthy performance with 91%, 96%, 93%, and 91% accuracy in predicting physical load, mental load, working memory, and attention, respectively. Additionally, we conducted SHAP (SHapley Additive exPlanations) analysis to identify the most critical features driving these predictions. Our findings contribute to understanding the overall cognitive state of a participant and effective data collection practices for future researchers, as well as provide insights for virtual reality developers. Developers can utilize these predictive approaches to adaptively optimize user experience in real-time and minimize cognitive strain, ultimately enhancing the effectiveness and usability of virtual reality applications.
随着虚拟现实（VR）技术的进步，虚拟环境中对多任务处理的需求不断升级。在沉浸式虚拟环境中协调多项任务带来了认知挑战，用户在执行多个并发任务时会遇到困难。这种现象凸显了注意力和工作记忆等认知功能的重要性，这些功能对于有效驾驭复杂的虚拟环境至关重要。除了注意力和工作记忆外，评估虚拟环境以及参与者执行的并发任务所引起的身心压力程度也是关键。虽然先前的研究主要集中在调查影响虚拟现实中注意力和工作记忆的因素，但仍需要更全面的方法来预测身心压力以及这些认知方面。这一差距激发了我们的研究，我们利用了一个开放数据集——VRWalking，其中包括眼睛和头部跟踪以及心率（HR）和皮肤电反应（GSR）等生理测量数据。VRwalking数据集具有带时间戳的关于身心负荷、工作记忆和注意力指标的标记数据。在我们的研究中，我们采用了简单的深度学习模型来预测这些标签，在预测身体负荷、心理负荷、工作记忆和注意力方面分别取得了91%、96%、93%和91%的显著准确率。此外，我们进行了SHAP（SHapley Additive exPlanations）分析，以确定推动这些预测的最关键特征。我们的研究结果有助于理解参与者的整体认知状态，并为未来的研究人员提供有效的数据收集方法，同时也为虚拟现实开发者提供了见解。开发者可以利用这些预测方法实时自适应地优化用户体验，并最大限度地减少认知压力，最终提高虚拟现实应用的有效性和可用性。
