Electro-Optics Engineering Department, School of Electrical and Computer Engineering, Ben Gurion University of the Negev, Beer Sheva, Israel.
Psychology Department, Ben Gurion University of the Negev, Beer Sheva, Israel.
Sci Rep. 2022 Jul 1;12(1):11188. doi: 10.1038/s41598-022-14808-4.
We describe a new method for remote emotional state assessment using multispectral face videos, and present our findings: unique transdermal, cardiovascular and spatiotemporal facial patterns associated with different emotional states. The method does not rely on stereotypical facial expressions but utilizes different wavelength sensitivities (visible spectrum, near-infrared, and long-wave infrared) to gauge correlates of autonomic nervous system activity spatially and temporally distributed across the human face (e.g., blood flow, hemoglobin concentration, and temperature). We conducted an experiment where 110 participants viewed 150 short emotion-eliciting videos and reported their emotional experience, while three cameras recorded facial videos with multiple wavelengths. Spatiotemporal multispectral features from the multispectral videos were used as inputs to a machine learning model that was able to classify participants' emotional state (i.e., amusement, disgust, fear, sexual arousal, or no emotion) with satisfactory results (average ROC AUC score of 0.75), while providing feature importance analysis that allows the examination of facial occurrences per emotional state. We discuss findings concerning the different spatiotemporal patterns associated with different emotional states as well as the different advantages of the current method over existing approaches to emotion detection.
我们描述了一种使用多光谱面部视频进行远程情绪状态评估的新方法，并呈现了我们的发现：与不同情绪状态相关的独特的经皮、心血管和时空面部模式。该方法不依赖于刻板的面部表情，而是利用不同波长的敏感性（可见光谱、近红外和长波红外）来衡量自主神经系统活动在面部空间和时间上的分布相关物（例如，血流、血红蛋白浓度和温度）。我们进行了一项实验，其中 110 名参与者观看了 150 个短情绪诱发视频，并报告了他们的情绪体验，同时三个摄像机记录了具有多个波长的面部视频。多光谱视频的时空多光谱特征被用作机器学习模型的输入，该模型能够以令人满意的结果（平均 ROC AUC 得分为 0.75）对参与者的情绪状态（即娱乐、厌恶、恐惧、性唤起或无情绪）进行分类，同时提供特征重要性分析，允许检查每个情绪状态的面部发生情况。我们讨论了与不同情绪状态相关的不同时空模式以及当前方法相对于现有情绪检测方法的不同优势的发现。
