School of Software, Henan University of Engineering, Zhengzhou, China.
Comput Intell Neurosci. 2022 Sep 28;2022:3518879. doi: 10.1155/2022/3518879. eCollection 2022.
Video emotion recognition has attracted increasing attention. Most existing approaches are based on the spatial features extracted from video frames. The context information and their relationships in videos are often ignored. Thus, the performance of existing approaches is restricted. In this study, we propose a sparse spatial-temporal emotion graph convolutional network-based video emotion recognition method (SE-GCN). For the spatial graph, the emotional relationship between any two emotion proposal regions is first calculated and the sparse spatial graph is constructed according to the emotional relationship. For the temporal graph, the emotional information contained in each emotion proposal region is first analyzed and the sparse temporal graph is constructed by using the emotion proposal regions with rich emotional cues. Then, the reasoning features of the emotional relationship are obtained by the spatial-temporal GCN. Finally, the features of the emotion proposal regions and the spatial-temporal relationship features are fused to recognize the video emotion. Extensive experiments are conducted on four challenging benchmark datasets, that is, MHED, HEIV, VideoEmotion-8, and Ekman-6. The experimental results demonstrate that the proposed method achieves state-of-the-art performance.
视频情感识别受到了越来越多的关注。现有的大多数方法都是基于从视频帧中提取的空间特征。视频中的上下文信息及其关系往往被忽略。因此，现有方法的性能受到限制。在这项研究中，我们提出了一种基于稀疏时空情感图卷积网络的视频情感识别方法（SE-GCN）。对于空间图，首先计算任意两个情感提议区域之间的情感关系，并根据情感关系构建稀疏空间图。对于时间图，首先分析每个情感提议区域中包含的情感信息，并使用具有丰富情感线索的情感提议区域构建稀疏时间图。然后，通过时空 GCN 获得情感关系的推理特征。最后，融合情感提议区域的特征和时空关系特征来识别视频情感。在四个具有挑战性的基准数据集 MHED、HEIV、VideoEmotion-8 和 Ekman-6 上进行了广泛的实验。实验结果表明，所提出的方法取得了最先进的性能。
