Pan Hang, Yang Hongling, Xie Lun, Wang Zhiliang
Department of Computer Science, Changzhi University, Changzhi, China.
School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing, China.
Front Neurosci. 2023 Jul 27;17:1216181. doi: 10.3389/fnins.2023.1216181. eCollection 2023.
Micro-expressions are facial muscle movements that hide genuine emotions. In response to the challenge of micro-expression low-intensity, recent studies have attempted to locate localized areas of facial muscle movement. However, this ignores the feature redundancy caused by the inaccurate locating of the regions of interest.
This paper proposes a novel multi-scale fusion visual attention network (MFVAN), which learns multi-scale local attention weights to mask regions of redundancy features. Specifically, this model extracts the multi-scale features of the apex frame in the micro-expression video clips by convolutional neural networks. The attention mechanism focuses on the weights of local region features in the multi-scale feature maps. Then, we mask operate redundancy regions in multi-scale features and fuse local features with high attention weights for micro-expression recognition. The self-supervision and transfer learning reduce the influence of individual identity attributes and increase the robustness of multi-scale feature maps. Finally, the multi-scale classification loss, mask loss, and removing individual identity attributes loss joint to optimize the model.
The proposed MFVAN method is evaluated on SMIC, CASME II, SAMM, and 3DB-Combined datasets that achieve state-of-the-art performance. The experimental results show that focusing on local at the multi-scale contributes to micro-expression recognition.
This paper proposed MFVAN model is the first to combine image generation with visual attention mechanisms to solve the combination challenge problem of individual identity attribute interference and low-intensity facial muscle movements. Meanwhile, the MFVAN model reveal the impact of individual attributes on the localization of local ROIs. The experimental results show that a multi-scale fusion visual attention network contributes to micro-expression recognition.
微表情是隐藏真实情感的面部肌肉运动。为应对微表情低强度的挑战，近期研究试图定位面部肌肉运动的局部区域。然而，这忽略了因感兴趣区域定位不准确而导致的特征冗余问题。
本文提出一种新颖的多尺度融合视觉注意力网络（MFVAN），该网络学习多尺度局部注意力权重以掩盖冗余特征区域。具体而言，此模型通过卷积神经网络提取微表情视频片段中顶点帧的多尺度特征。注意力机制聚焦于多尺度特征图中局部区域特征的权重。然后，我们在多尺度特征中对冗余区域进行掩码操作，并融合具有高注意力权重的局部特征用于微表情识别。自监督和迁移学习减少了个体身份属性的影响，增强了多尺度特征图的鲁棒性。最后，多尺度分类损失、掩码损失和去除个体身份属性损失联合起来优化模型。
所提出的MFVAN方法在SMIC、CASME II、SAMM和3DB组合数据集上进行评估，取得了领先的性能。实验结果表明，在多尺度上关注局部有助于微表情识别。
本文提出的MFVAN模型首次将图像生成与视觉注意力机制相结合，以解决个体身份属性干扰和低强度面部肌肉运动的组合挑战问题。同时，MFVAN模型揭示了个体属性对局部感兴趣区域定位的影响。实验结果表明，多尺度融合视觉注意力网络有助于微表情识别。
