Javed Hifza, Wang Weinan, Usman Affan Bin, Jamali Nawid
Honda Research Institute USA, Inc., San Jose, CA, United States.
Department of Electrical and Computer Engineering, Rutgers University, Piscataway, NJ, United States.
Front Robot AI. 2024 Nov 28;11:1410957. doi: 10.3389/frobt.2024.1410957. eCollection 2024.
Social mediator robots have shown potential in facilitating human interactions by improving communication, fostering relationships, providing support, and promoting inclusivity. However, for these robots to effectively shape human interactions, they must understand the intricacies of interpersonal dynamics. This necessitates models of human understanding that capture interpersonal states and the relational affect arising from interactions. Traditional affect recognition methods, primarily focus on individual affect, and may fall short in capturing interpersonal dynamics crucial for social mediation. To address this gap, we propose a multimodal, multi-perspective model of relational affect, utilizing a conversational dataset collected in uncontrolled settings. Our model extracts features from audiovisual data to capture affective behaviors indicative of relational affect. By considering the interpersonal perspectives of both interactants, our model predicts relational affect, enabling real-time understanding of evolving interpersonal dynamics. We discuss our model's utility for social mediation applications and compare it with existing approaches, highlighting its advantages for real-world applicability. Despite the complexity of human interactions and subjective nature of affect ratings, our model demonstrates early capabilities to enable proactive intervention in negative interactions, enhancing neutral exchanges, and respecting positive dialogues. We discuss implications for real-world deployment and highlight the limitations of current work. Our work represents a step towards developing computational models of relational affect tailored for real-world social mediation, offering insights into effective mediation strategies for social mediator robots.
社交调解机器人已展现出通过改善沟通、促进关系、提供支持和推动包容性来促进人际互动的潜力。然而，要使这些机器人有效地塑造人际互动，它们必须理解人际动态的复杂性。这就需要能够捕捉人际状态和互动中产生的关系情感的人类理解模型。传统的情感识别方法主要关注个体情感，可能无法捕捉对社会调解至关重要的人际动态。为了弥补这一差距，我们利用在不受控制的环境中收集的对话数据集，提出了一种多模态、多视角的关系情感模型。我们的模型从视听数据中提取特征，以捕捉指示关系情感的情感行为。通过考虑互动双方的人际视角，我们的模型预测关系情感，从而实现对不断演变的人际动态的实时理解。我们讨论了我们的模型在社会调解应用中的效用，并将其与现有方法进行比较，突出其在实际应用中的优势。尽管人际互动复杂且情感评分具有主观性，但我们的模型展示了早期能力，能够对负面互动进行主动干预，加强中性交流，并尊重积极对话。我们讨论了对实际部署的影响，并强调了当前工作的局限性。我们的工作代表了朝着为现实世界的社会调解量身定制关系情感计算模型迈出的一步，为社交调解机器人的有效调解策略提供了见解。
