Castellano Giovanna, De Carolis Berardina, Macchiarulo Nicola
Department of Computer Science, University of Bari, Bari, Italy.
Exprivia S.p.A., Molfetta, Italy.
Multimed Tools Appl. 2023;82(9):12751-12769. doi: 10.1007/s11042-022-14050-0. Epub 2022 Oct 22.
People use various nonverbal communicative channels to convey emotions, among which facial expressions are considered the most important ones. Thus, automatic Facial Expression Recognition (FER) is a fundamental task to increase the perceptive skills of computers, especially in human-computer interaction. Like humans, state-of-art FER systems are able to recognize emotions from the entire face of a person. However, the COVID-19 pandemic has imposed a massive use of face masks that help in preventing infection but may hamper social communication and make the recognition of facial expressions a very challenging task due to facial occlusion. In this paper we propose a FER system capable to recognize emotions from masked faces. The system checks for the presence of a mask on the face image and, in case of mask detection, it extracts the eyes region and recognizes the emotion only considering that portion of the face. The effectiveness of the developed FER system was tested in recognizing emotions and their valence only from the eyes region and comparing the results when considering the entire face. As it was expected, emotions that are related mainly to the mouth region (e.g., disgust) are barely recognized, while positive emotions are better identified by considering only the eyes region. Moreover, we compared the results of our FER system to the human annotation of emotions on masked faces. We found out that the FER system outperforms the human annotation, thus showing that the model is able to learn proper features for each emotion leveraging only the eyes region.
人们使用各种非语言交流渠道来传达情感，其中面部表情被认为是最重要的。因此，自动面部表情识别（FER）是提高计算机感知能力的一项基本任务，尤其是在人机交互方面。与人类一样，先进的FER系统能够从人的整个面部识别情感。然而，新冠疫情导致口罩被大量使用，口罩有助于预防感染，但可能会妨碍社交沟通，并且由于面部遮挡，使得面部表情识别成为一项极具挑战性的任务。在本文中，我们提出了一种能够从戴口罩的面部识别情感的FER系统。该系统会检查面部图像上是否有口罩，若检测到口罩，它会提取眼睛区域，并仅考虑面部的该部分来识别情感。所开发的FER系统的有效性通过仅从眼睛区域识别情感及其效价，并与考虑整个面部时的结果进行比较来进行测试。正如预期的那样，主要与嘴部区域相关的情感（如厌恶）几乎无法识别，而仅考虑眼睛区域时，积极情感能被更好地识别。此外，我们将FER系统的结果与人类对戴口罩面部的情感标注进行了比较。我们发现FER系统的表现优于人类标注，从而表明该模型仅利用眼睛区域就能为每种情感学习到合适的特征。
