Washington Peter, Kalantarian Haik, Kent John, Husic Arman, Kline Aaron, Leblanc Emilie, Hou Cathy, Mutlu Onur Cezmi, Dunlap Kaitlyn, Penev Yordan, Varma Maya, Stockham Nate Tyler, Chrisman Brianna, Paskov Kelley, Sun Min Woo, Jung Jae-Yoon, Voss Catalin, Haber Nick, Wall Dennis Paul
Departments of Pediatrics (Systems Medicine) and Biomedical Data Science, Stanford University, Stanford, CA, United States.
JMIR Pediatr Parent. 2022 Apr 8;5(2):e26760. doi: 10.2196/26760.
Automated emotion classification could aid those who struggle to recognize emotions, including children with developmental behavioral conditions such as autism. However, most computer vision emotion recognition models are trained on adult emotion and therefore underperform when applied to child faces.
We designed a strategy to gamify the collection and labeling of child emotion-enriched images to boost the performance of automatic child emotion recognition models to a level closer to what will be needed for digital health care approaches.
We leveraged our prototype therapeutic smartphone game, GuessWhat, which was designed in large part for children with developmental and behavioral conditions, to gamify the secure collection of video data of children expressing a variety of emotions prompted by the game. Independently, we created a secure web interface to gamify the human labeling effort, called HollywoodSquares, tailored for use by any qualified labeler. We gathered and labeled 2155 videos, 39,968 emotion frames, and 106,001 labels on all images. With this drastically expanded pediatric emotion-centric database (>30 times larger than existing public pediatric emotion data sets), we trained a convolutional neural network (CNN) computer vision classifier of happy, sad, surprised, fearful, angry, disgust, and neutral expressions evoked by children.
The classifier achieved a 66.9% balanced accuracy and 67.4% F1-score on the entirety of the Child Affective Facial Expression (CAFE) as well as a 79.1% balanced accuracy and 78% F1-score on CAFE Subset A, a subset containing at least 60% human agreement on emotions labels. This performance is at least 10% higher than all previously developed classifiers evaluated against CAFE, the best of which reached a 56% balanced accuracy even when combining "anger" and "disgust" into a single class.
This work validates that mobile games designed for pediatric therapies can generate high volumes of domain-relevant data sets to train state-of-the-art classifiers to perform tasks helpful to precision health efforts.
自动情感分类可以帮助那些难以识别情感的人，包括患有自闭症等发育行为疾病的儿童。然而，大多数计算机视觉情感识别模型是基于成人情感进行训练的，因此应用于儿童面部时表现不佳。
我们设计了一种策略，将丰富儿童情感的图像的收集和标注游戏化，以将自动儿童情感识别模型的性能提升至更接近数字医疗方法所需的水平。
我们利用了我们的原型治疗性智能手机游戏“猜猜是什么”（GuessWhat），该游戏在很大程度上是为患有发育和行为疾病的儿童设计的，将儿童在游戏引发下表达各种情感的视频数据的安全收集游戏化。我们独立创建了一个安全的网络界面“好莱坞方块”（HollywoodSquares），将人工标注工作游戏化，供任何合格标注人员使用。我们收集并标注了2155个视频、39968个情感帧以及所有图像上的106001个标签。利用这个大幅扩展的以儿童情感为中心的数据库（比现有的公共儿童情感数据集大30倍以上），我们训练了一个卷积神经网络（CNN）计算机视觉分类器，用于识别儿童引发的快乐、悲伤、惊讶、恐惧、愤怒、厌恶和中性表情。
该分类器在整个儿童情感面部表情（CAFE）数据集上实现了66.9%的平衡准确率和67.4%的F1分数，在CAFE子集A上实现了79.1%的平衡准确率和78%的F1分数，CAFE子集A是一个在情感标签上至少有60%人类一致性的子集。这一性能比之前针对CAFE评估的所有分类器至少高出10%，其中最好的分类器即使将“愤怒”和“厌恶”合并为一个类别，也仅达到了56%的平衡准确率。
这项工作验证了为儿科治疗设计的手机游戏可以生成大量与领域相关的数据集，以训练先进的分类器来执行有助于精准医疗工作的任务。
