Andreu-Perez Ana R, Kiani Mehrin, Andreu-Perez Javier, Reddy Pratusha, Andreu-Abela Jaime, Pinto Maria, Izzetoglu Kurtulus
Faculty of Communication and Documentation, University of Granada, 18071 Granada, Spain.
Centre for Computational Intelligence, School of Computer Science and Electronic Engineering, University of Essex, Colchester CO4 3SQ, UK.
Brain Sci. 2021 Jan 14;11(1):106. doi: 10.3390/brainsci11010106.
With an increase in consumer demand of video gaming entertainment, the game industry is exploring novel ways of game interaction such as providing direct interfaces between the game and the gamers' cognitive or affective responses. In this work, gamer's brain activity has been imaged using functional near infrared spectroscopy (fNIRS) whilst they watch video of a video game (League of Legends) they play. A video of the face of the participants is also recorded for each of a total of 15 trials where a trial is defined as watching a gameplay video. From the data collected, i.e., gamer's fNIRS data in combination with emotional state estimation from gamer's facial expressions, the expertise level of the gamers has been decoded per trial in a multi-modal framework comprising of unsupervised deep feature learning and classification by state-of-the-art models. The best tri-class classification accuracy is obtained using a cascade of random convolutional kernel transform (ROCKET) feature extraction method and deep classifier at 91.44%. This is the first work that aims at decoding expertise level of gamers using non-restrictive and portable technologies for brain imaging, and emotional state recognition derived from gamers' facial expressions. This work has profound implications for novel designs of future human interactions with video games and brain-controlled games.
随着消费者对视频游戏娱乐需求的增加，游戏行业正在探索新颖的游戏交互方式，例如在游戏与玩家的认知或情感反应之间提供直接接口。在这项工作中，当玩家观看他们所玩视频游戏（《英雄联盟》）的视频时，使用功能近红外光谱（fNIRS）对玩家的大脑活动进行了成像。在总共15次试验中，每次试验定义为观看游戏玩法视频时，还记录了参与者面部的视频。从收集到的数据，即玩家的fNIRS数据与根据玩家面部表情进行的情绪状态估计相结合，在一个由无监督深度特征学习和先进模型分类组成的多模态框架中，每次试验对玩家的专业水平进行了解码。使用级联随机卷积核变换（ROCKET）特征提取方法和深度分类器获得了最佳的三分类准确率，为91.44%。这是第一项旨在使用非限制性和便携式脑成像技术以及从玩家面部表情得出的情绪状态识别来解码玩家专业水平的工作。这项工作对未来人类与视频游戏和脑控游戏的新颖设计具有深远意义。
