Graduate School of Information Science and Technology, Hokkaido University, N-14, W-9, Kita-ku, Sapporo, Hokkaido 060-0814, Japan.
Office of Institutional Research, Hokkaido University, N-8, W-5, Kita-ku, Sapporo, Hokkaido 060-0808, Japan.
Sensors (Basel). 2020 Apr 10;20(7):2146. doi: 10.3390/s20072146.
The paper proposes a method of visual attention-based emotion classification through eye gaze analysis. Concretely, tensor-based emotional category classification via visual attention-based heterogeneous convolutional neural network (CNN) feature fusion is proposed. Based on the relationship between human emotions and changes in visual attention with time, the proposed method performs new gaze-based image representation that is suitable for reflecting the characteristics of the changes in visual attention with time. Furthermore, since emotions evoked in humans are closely related to objects in images, our method uses a CNN model to obtain CNN features that can represent their characteristics. For improving the representation ability to the emotional categories, we extract multiple CNN features from our novel gaze-based image representation and enable their fusion by constructing a novel tensor consisting of these CNN features. Thus, this tensor construction realizes the visual attention-based heterogeneous CNN feature fusion. This is the main contribution of this paper. Finally, by applying logistic tensor regression with general tensor discriminant analysis to the newly constructed tensor, the emotional category classification becomes feasible. Since experimental results show that the proposed method enables the emotional category classification with the F1-measure of approximately 0.6, and about 10% improvement can be realized compared to comparative methods including state-of-the-art methods, the effectiveness of the proposed method is verified.
本文提出了一种基于视觉注意的情感分类方法，通过眼动分析实现。具体来说，本文提出了一种基于张量的情感分类方法，通过基于视觉注意的异构卷积神经网络（CNN）特征融合实现。基于人类情感与随时间变化的视觉注意之间的关系，本文提出了一种新的基于注视的图像表示方法，该方法适合反映随时间变化的视觉注意特征。此外，由于人类产生的情感与图像中的对象密切相关，我们的方法使用 CNN 模型来获取能够代表其特征的 CNN 特征。为了提高对情感类别的表示能力，我们从新的基于注视的图像表示中提取多个 CNN 特征，并通过构建包含这些 CNN 特征的新张量来实现它们的融合。因此，这种张量构建实现了基于视觉注意的异构 CNN 特征融合。这是本文的主要贡献。最后，通过将一般张量判别分析的逻辑张量回归应用于新构建的张量，实现了情感类别分类。由于实验结果表明，与包括最新方法在内的对比方法相比，该方法能够实现约 0.6 的 F1 度量的情感类别分类，并且可以实现约 10%的改进，因此验证了该方法的有效性。
