Saffaryazdi Nastaran, Wasim Syed Talal, Dileep Kuldeep, Nia Alireza Farrokhi, Nanayakkara Suranga, Broadbent Elizabeth, Billinghurst Mark
Empathic Computing Laboratory, Auckland Bioengineering Institute, The University of Auckland, Auckland, New Zealand.
Augmented Human Laboratory, Auckland Bioengineering Institute, The University of Auckland, Auckland, New Zealand.
Front Psychol. 2022 Jun 28;13:864047. doi: 10.3389/fpsyg.2022.864047. eCollection 2022.
Emotions are multimodal processes that play a crucial role in our everyday lives. Recognizing emotions is becoming more critical in a wide range of application domains such as healthcare, education, human-computer interaction, Virtual Reality, intelligent agents, entertainment, and more. Facial macro-expressions or intense facial expressions are the most common modalities in recognizing emotional states. However, since facial expressions can be voluntarily controlled, they may not accurately represent emotional states. Earlier studies have shown that facial micro-expressions are more reliable than facial macro-expressions for revealing emotions. They are subtle, involuntary movements responding to external stimuli that cannot be controlled. This paper proposes using facial micro-expressions combined with brain and physiological signals to more reliably detect underlying emotions. We describe our models for measuring arousal and valence levels from a combination of facial micro-expressions, Electroencephalography (EEG) signals, galvanic skin responses (GSR), and Photoplethysmography (PPG) signals. We then evaluate our model using the DEAP dataset and our own dataset based on a subject-independent approach. Lastly, we discuss our results, the limitations of our work, and how these limitations could be overcome. We also discuss future directions for using facial micro-expressions and physiological signals in emotion recognition.
情绪是多模态过程，在我们的日常生活中起着至关重要的作用。在医疗保健、教育、人机交互、虚拟现实、智能代理、娱乐等广泛的应用领域中，识别情绪变得越来越重要。面部宏观表情或强烈的面部表情是识别情绪状态最常见的模态。然而，由于面部表情可以被自主控制，它们可能无法准确地代表情绪状态。早期研究表明，面部微表情在揭示情绪方面比面部宏观表情更可靠。它们是对无法控制的外部刺激做出的细微、不自主的动作。本文提出结合面部微表情与大脑和生理信号来更可靠地检测潜在情绪。我们描述了从面部微表情、脑电图（EEG）信号、皮肤电反应（GSR）和光电容积脉搏波描记法（PPG）信号的组合中测量唤醒和效价水平的模型。然后，我们基于独立于受试者的方法，使用DEAP数据集和我们自己的数据集对我们的模型进行评估。最后，我们讨论了我们的结果、工作的局限性以及如何克服这些局限性。我们还讨论了在情绪识别中使用面部微表情和生理信号的未来方向。
