Campo Adriaan, Michałko Aleksandra, Van Kerrebroeck Bavo, Stajic Boris, Pokric Maja, Leman Marc
Department of Art, Music and Theatre Sciences, Faculty of Arts and Philosophy, Institute for Psychoacoustics and Electronic Music (IPEM), Ghent University, Miriam Makebaplein 1, B-9000, Gent België, Belgium.
ARVRtech, Antona Čehova 1, Novi Sad, Serbia.
Comput Human Behav. 2023 Sep;146:107810. doi: 10.1016/j.chb.2023.107810.
The acquisition of advanced gestures is a challenge in various domains of proficient sensorimotor performance. For example, orchestral violinists must move in sync with the lead violinist's gestures. To help train these gestures, an educational music play-back system was developed using a HoloLens 2 simulated AR environment and an avatar representation of the lead violinist. This study aimed to investigate the impact of using a 2D or 3D representation of the lead violinist's avatar on students' learning experience in the AR environment. To assess the learning outcome, the study employed a longitudinal experiment design, in which eleven participants practiced two pieces of music in four trials, evenly spaced over a month. Participants were asked to mimic the avatar's gestures as closely as possible when it came to using the bow, including bowing, articulations, and dynamics. The study compared the similarities between the avatar's gestures and those of the participants at the biomechanical level, using motion capture measurements, as well as the smoothness of the participants' movements. Additionally, presence and perceived difficulty were assessed using questionnaires. The results suggest that using a 3D representation of the avatar leads to better gesture resemblance and a higher experience of presence compared to a 2D representation. The 2D representation, however, showed a learning effect, but this was not observed in the 3D condition. The findings suggest that the 3D condition benefits from stereoscopic information that enhances spatial cognition, making it more effective in relation to sensorimotor performance. Overall, the 3D condition had a greater impact on performance than on learning. This work concludes with recommendations for future efforts directed towards AR-based advanced gesture training to address the challenges related to measurement methodology and participants' feedback on the AR application.
掌握高级手势是熟练的感觉运动表现各个领域中的一项挑战。例如,管弦乐队的小提琴手必须与首席小提琴手的手势同步移动。为了帮助训练这些手势,利用HoloLens 2模拟增强现实(AR)环境和首席小提琴手的虚拟形象开发了一种教育性音乐回放系统。本研究旨在调查使用首席小提琴手虚拟形象的二维或三维表示对学生在AR环境中学习体验的影响。为了评估学习成果,该研究采用了纵向实验设计,其中11名参与者在四个试验中练习两首乐曲,试验在一个月内均匀分布。在使用琴弓时,要求参与者尽可能紧密地模仿虚拟形象的手势,包括运弓、发音和力度变化。该研究使用动作捕捉测量方法,在生物力学层面比较了虚拟形象的手势与参与者手势之间的相似性,以及参与者动作的流畅性。此外,还通过问卷调查评估了临场感和感知难度。结果表明,与二维表示相比,使用三维虚拟形象能带来更好的手势相似度和更高的临场感体验。然而,二维表示显示出学习效果,但在三维条件下未观察到这种效果。研究结果表明,三维条件受益于增强空间认知的立体信息,使其在感觉运动表现方面更有效。总体而言;三维条件对表现的影响大于对学习的影响。这项工作最后针对未来基于AR的高级手势训练提出了建议,以应对与测量方法和参与者对AR应用的反馈相关的挑战。