Lin Chujun, Bulls Landry S, Tepfer Lindsey J, Vyas Amisha D, Thornton Mark A
Department of Psychological and Brain Sciences, Dartmouth College, Hanover, NH USA.
Affect Sci. 2023 Aug 25;4(3):550-562. doi: 10.1007/s42761-023-00215-z. eCollection 2023 Sep.
People express their own emotions and perceive others' emotions via a variety of channels, including facial movements, body gestures, vocal prosody, and language. Studying these channels of affective behavior offers insight into both the experience and perception of emotion. Prior research has predominantly focused on studying individual channels of affective behavior in isolation using tightly controlled, non-naturalistic experiments. This approach limits our understanding of emotion in more naturalistic contexts where different channels of information tend to interact. Traditional methods struggle to address this limitation: manually annotating behavior is time-consuming, making it infeasible to do at large scale; manually selecting and manipulating stimuli based on hypotheses may neglect unanticipated features, potentially generating biased conclusions; and common linear modeling approaches cannot fully capture the complex, nonlinear, and interactive nature of real-life affective processes. In this methodology review, we describe how deep learning can be applied to address these challenges to advance a more naturalistic affective science. First, we describe current practices in affective research and explain why existing methods face challenges in revealing a more naturalistic understanding of emotion. Second, we introduce deep learning approaches and explain how they can be applied to tackle three main challenges: quantifying naturalistic behaviors, selecting and manipulating naturalistic stimuli, and modeling naturalistic affective processes. Finally, we describe the limitations of these deep learning methods, and how these limitations might be avoided or mitigated. By detailing the promise and the peril of deep learning, this review aims to pave the way for a more naturalistic affective science.
人们通过多种渠道表达自己的情感并感知他人的情感，这些渠道包括面部动作、身体姿势、语音韵律和语言。研究这些情感行为渠道有助于深入了解情感体验和感知。先前的研究主要集中在使用严格控制的非自然主义实验来孤立地研究情感行为的各个渠道。这种方法限制了我们在更自然主义的情境中对情感的理解，在这些情境中不同的信息渠道往往会相互作用。传统方法难以解决这一局限性：手动标注行为耗时费力，大规模进行不可行；基于假设手动选择和操纵刺激可能会忽略意外特征，从而可能得出有偏差的结论；常见的线性建模方法无法完全捕捉现实生活中情感过程的复杂、非线性和交互性质。在本方法综述中，我们描述了深度学习如何应用于应对这些挑战，以推动更自然主义的情感科学发展。首先，我们描述情感研究中的当前实践，并解释为什么现有方法在揭示对情感更自然主义的理解方面面临挑战。其次，我们介绍深度学习方法，并解释它们如何应用于应对三个主要挑战：量化自然主义行为、选择和操纵自然主义刺激以及对自然主义情感过程进行建模。最后，我们描述这些深度学习方法的局限性，以及如何避免或减轻这些局限性。通过详细阐述深度学习的前景和风险，本综述旨在为更自然主义的情感科学铺平道路。
