Zeng Haipeng, Wang Xingbo, Wu Aoyu, Wang Yong, Li Quan, Endert Alex, Qu Huamin
IEEE Trans Vis Comput Graph. 2020 Jan;26(1):927-937. doi: 10.1109/TVCG.2019.2934656. Epub 2019 Aug 20.
Emotions play a key role in human communication and public presentations. Human emotions are usually expressed through multiple modalities. Therefore, exploring multimodal emotions and their coherence is of great value for understanding emotional expressions in presentations and improving presentation skills. However, manually watching and studying presentation videos is often tedious and time-consuming. There is a lack of tool support to help conduct an efficient and in-depth multi-level analysis. Thus, in this paper, we introduce EmoCo, an interactive visual analytics system to facilitate efficient analysis of emotion coherence across facial, text, and audio modalities in presentation videos. Our visualization system features a channel coherence view and a sentence clustering view that together enable users to obtain a quick overview of emotion coherence and its temporal evolution. In addition, a detail view and word view enable detailed exploration and comparison from the sentence level and word level, respectively. We thoroughly evaluate the proposed system and visualization techniques through two usage scenarios based on TED Talk videos and interviews with two domain experts. The results demonstrate the effectiveness of our system in gaining insights into emotion coherence in presentations.
情感在人际交流和公众演讲中起着关键作用。人类情感通常通过多种方式表达。因此，探索多模态情感及其连贯性对于理解演讲中的情感表达和提高演讲技巧具有重要价值。然而，手动观看和研究演讲视频往往既乏味又耗时。缺乏工具支持来帮助进行高效且深入的多层次分析。因此，在本文中，我们介绍了EmoCo，这是一个交互式视觉分析系统，用于促进对演讲视频中面部、文本和音频模态的情感连贯性进行高效分析。我们的可视化系统具有通道连贯性视图和句子聚类视图，两者共同使用能让用户快速了解情感连贯性及其时间演变。此外，细节视图和单词视图分别能从句子层面和单词层面进行详细探索和比较。我们通过基于TED演讲视频的两个使用场景以及与两位领域专家的访谈，对所提出的系统和可视化技术进行了全面评估。结果证明了我们的系统在洞察演讲中情感连贯性方面的有效性。
