Kargarandehkordi Ali, Kaisti Matti, Washington Peter
Information and Computer Sciences Department, University of Hawai'i at Manoa, Honolulu, HI 96822, USA.
Department of Computing, University of Turku, 20014 Turku, Finland.
Appl Sci (Basel). 2024 Feb 2;14(4). doi: 10.3390/app14041337. Epub 2024 Feb 6.
Our study delves into the concept of model personalization in emotion recognition, moving away from the one-size-fits-all approach. We conducted a series of experiments using the Emognition dataset, comprising physiological and video data of human subjects expressing various emotions, to investigate a personalized approach to affective computing. For the 10 individuals in the dataset with a sufficient representation of at least two ground truth emotion labels, we trained a personalized version of three classical ML models (k-nearest neighbors, random forests, and a dense neural network) on a set of 51 features extracted from each video frame. We ensured that all the frames used to train the models occurred earlier in the video than the frames used to test the model. We measured the importance of each facial feature for all the personalized models and observed differing ranked lists of the top features across the subjects, highlighting the need for model personalization. We then compared the personalized models against a generalized model trained using data from all 10 subjects. The mean F1 scores for the personalized models, specifically for the k-nearest neighbors, random forest, and dense neural network, were 90.48%, 92.66%, and 86.40%, respectively. In contrast, the mean F1 scores for the generic models, using the same ML techniques, were 88.55%, 91.78% and 80.42%, respectively, when trained on data from various human subjects and evaluated using the same test set. The personalized models outperformed the generalized models for 7 out of the 10 subjects. The PCA analyses on the remaining three subjects revealed relatively little facial configuration differences across the emotion labels within each subject, suggesting that personalized ML will fail when the variation among data points within a subject's data is too low. This preliminary feasibility study demonstrates the potential as well as the ongoing challenges with implementing personalized models which predict highly subjective outcomes like emotion.
我们的研究深入探讨了情感识别中模型个性化的概念，摒弃了一刀切的方法。我们使用Emognition数据集进行了一系列实验，该数据集包含表达各种情感的人类受试者的生理和视频数据，以研究情感计算的个性化方法。对于数据集中至少有两个真实情感标签充分表示的10个人，我们在从每个视频帧中提取的一组51个特征上训练了三种经典机器学习模型（k近邻、随机森林和密集神经网络）的个性化版本。我们确保用于训练模型的所有帧在视频中比用于测试模型的帧出现得更早。我们测量了所有个性化模型中每个面部特征的重要性，并观察到不同受试者的顶级特征排名列表不同，突出了模型个性化的必要性。然后，我们将个性化模型与使用所有10个受试者的数据训练的通用模型进行了比较。个性化模型的平均F1分数，具体来说，k近邻、随机森林和密集神经网络分别为90.48%、92.66%和86.40%。相比之下，使用相同机器学习技术的通用模型在使用来自不同人类受试者的数据进行训练并使用相同测试集进行评估时，平均F1分数分别为88.55%、91.78%和80.42%。10名受试者中有7名的个性化模型优于通用模型。对其余三名受试者的主成分分析表明，每个受试者内不同情感标签之间的面部配置差异相对较小，这表明当受试者数据内的数据点变化过低时，个性化机器学习将失败。这项初步可行性研究展示了实施预测情感等高度主观结果的个性化模型的潜力以及持续存在的挑战。
