Wu Chenyan, Davaasuren Dolzodmaa, Shafir Tal, Tsachor Rachelle, Wang James Z
Data Science and Artificial Intelligence Area, College of Information Sciences and Technology, The Pennsylvania State University, University Park, PA 16802, USA.
The Emili Sagol Creative Arts Therapies Research Center, University of Haifa, Haifa 3498838, Israel.
Patterns (N Y). 2023 Aug 22;4(10):100816. doi: 10.1016/j.patter.2023.100816. eCollection 2023 Oct 13.
Bodily expressed emotion understanding (BEEU) aims to automatically recognize human emotional expressions from body movements. Psychological research has demonstrated that people often move using specific motor elements to convey emotions. This work takes three steps to integrate human motor elements to study BEEU. First, we introduce BoME (body motor elements), a highly precise dataset for human motor elements. Second, we apply baseline models to estimate these elements on BoME, showing that deep learning methods are capable of learning effective representations of human movement. Finally, we propose a dual-source solution to enhance the BEEU model with the BoME dataset, which trains with both motor element and emotion labels and simultaneously produces predictions for both. Through experiments on the BoLD in-the-wild emotion understanding benchmark, we showcase the significant benefit of our approach. These results may inspire further research utilizing human motor elements for emotion understanding and mental health analysis.
身体表达情绪理解（BEEU）旨在从身体动作中自动识别人类的情绪表达。心理学研究表明，人们常常通过特定的运动元素来传达情绪。这项工作分三步整合人类运动元素以研究身体表达情绪理解。首先，我们引入了BoME（身体运动元素），这是一个关于人类运动元素的高精度数据集。其次，我们应用基线模型在BoME上估计这些元素，表明深度学习方法能够学习人类运动的有效表征。最后，我们提出了一种双源解决方案，用BoME数据集增强身体表达情绪理解模型，该模型同时使用运动元素和情绪标签进行训练，并同时对两者进行预测。通过在BoLD野外情绪理解基准上的实验，我们展示了我们方法的显著优势。这些结果可能会激发利用人类运动元素进行情绪理解和心理健康分析的进一步研究。
