Faculty of Environment and Information Sciences, Yokohama National University, Yokohama, Kanagawa, Japan.
PLoS One. 2024 Oct 30;19(10):e0313029. doi: 10.1371/journal.pone.0313029. eCollection 2024.
Facial expression recognition (FER) is significantly influenced by the cultural background (CB) of observers and the masking conditions of the target face. This study aimed to clarify these factors' impact on FER, particularly in machine-learning datasets, increasingly used in human-computer interaction and automated systems. We conducted an FER experiment with East Asian participants and compared the results with the FERPlus dataset, evaluated by Western raters. Our novel analysis approach focused on variability between images and participants within a "majority" category and the eye-opening rate of target faces, providing a deeper understanding of FER processes. Notable findings were differences in "fear" perception between East Asians and Westerners, with East Asians more likely to interpret "fear" as "surprise." Masking conditions significantly affected emotion categorization, with "fear" perceived by East Asians for non-masked faces interpreted as "surprise" for masked faces. Then, the emotion labels were perceived as different emotions across categories in the masking condition, rather than simply lower recognition rates or confusion as in existing studies. Additionally, "sadness" perceived by Westerners was often interpreted as "disgust" by East Asians. These results suggest that one-to-one network learning models, commonly trained using majority labels, might overlook important minority response information, potentially leading to biases in automated FER systems. In conclusion, FER dataset characteristics differ depending on the target face's masking condition and the diversity among evaluation groups. This study highlights the need to consider these factors in machine-learning-based FER that relies on human-judged labels, to contribute to the development of more nuanced and fair automated FER systems. Our findings emphasize the novelty of our approach compared to existing studies and the importance of incorporating a broader range of human variability in FER research, setting the stage for future evaluations of machine learning classifiers on similar data.
面部表情识别（FER）受到观察者的文化背景（CB）和目标面部的掩蔽条件的显著影响。本研究旨在阐明这些因素对 FER 的影响，特别是在机器学习数据集方面，这些数据集越来越多地用于人机交互和自动化系统。我们进行了一项东亚参与者的 FER 实验，并将结果与西方评估者评估的 FERPlus 数据集进行了比较。我们的新分析方法侧重于“多数”类别内的图像和参与者之间的变异性以及目标面部的睁眼率，从而深入了解 FER 过程。值得注意的发现是东亚人和西方人对“恐惧”的感知存在差异，东亚人更有可能将“恐惧”解释为“惊讶”。掩蔽条件显著影响情绪分类，对于非掩蔽面孔，东亚人感知到的“恐惧”在掩蔽面孔时被解释为“惊讶”。然后，在掩蔽条件下，情绪标签被感知为不同类别的不同情绪，而不是像现有研究那样只是识别率降低或混淆。此外，西方人感知到的“悲伤”往往被东亚人解释为“厌恶”。这些结果表明，常用多数标签训练的一对一网络学习模型可能会忽略重要的少数响应信息，从而导致自动化 FER 系统出现偏差。总之，FER 数据集的特征取决于目标面部的掩蔽条件和评估群体的多样性。本研究强调了在基于机器学习的 FER 中考虑这些因素的必要性，这有助于开发更细致和公平的自动化 FER 系统。我们的研究结果强调了与现有研究相比，我们的方法的新颖性以及在 FER 研究中纳入更广泛的人类变异性的重要性，为在类似数据上评估机器学习分类器奠定了基础。
