IEEE Trans Cybern. 2018 Jul;48(7):2203-2214. doi: 10.1109/TCYB.2017.2786309.
Visible facial images provide geometric and appearance patterns of facial expressions and are sensitive to illumination changes. Thermal facial images record facial temperature distribution and are robust to light conditions. Therefore, expression recognition is enhanced by visible and thermal image fusion. In most cases, only visible images are available due to the widespread popularity of visible cameras and the high cost of thermal cameras. Thus, we propose a novel visible expression recognition method by using thermal infrared (IR) data as privileged information, which is only available during training. Specifically, we first learn a deep model for visible images and thermal images. Then we use the learned feature representations to train support vector machine (SVM) classifiers for expression classification. We jointly refine the deep models as well as the SVM classifiers for both thermal images and visible images by imposing the constraint that the outputs of the SVM classifiers from two views are similar. Thermal IR images during training are then exploited to construct better facial representations and expression classifiers from visible images. We extend the proposed thermal augmented expression recognition method for partially unpaired data, acknowledging that visible images and thermal images maybe not be recorded synchronously. Experimental resulton the MAHNOB laughter database demonstrate that the proposed thermal augmented expression recognition method can effectively exploit thermal IR images' supplementary role for visible facial expression recognition during training to obtain better facial representations and a better visible expression classifier. The proposed thermal augmented expression recognition method achieves state-of-the-art expression recognition performance for both paired and unpaired facial images.
可见面部图像提供了面部表情的几何和外观模式，并且对光照变化敏感。热面部图像记录面部温度分布，对光照条件具有鲁棒性。因此，通过可见和热图像融合可以增强表情识别。在大多数情况下，由于可见摄像机的广泛普及和热摄像机的高成本，只有可见图像可用。因此，我们提出了一种新颖的可见表情识别方法，该方法使用热红外（IR）数据作为特权信息，仅在训练期间可用。具体来说，我们首先学习可见图像和热图像的深度模型。然后，我们使用学习到的特征表示来训练支持向量机（SVM）分类器进行表情分类。我们通过施加 SVM 分类器的输出来自两个视图相似的约束，共同改进热图像和可见图像的深度模型以及 SVM 分类器。然后，利用训练期间的热红外图像从可见图像中构建更好的面部表示和表情分类器。我们将提出的热增强表情识别方法扩展到部分非配对数据，承认可见图像和热图像可能不是同步记录的。在 MAHNOB 笑声数据库上的实验结果表明，所提出的热增强表情识别方法可以有效地利用热红外图像在训练期间对可见面部表情识别的补充作用，从而获得更好的面部表示和更好的可见表情分类器。所提出的热增强表情识别方法在配对和非配对面部图像的表情识别性能方面均达到了最新水平。
