IEEE Trans Image Process. 2016 Dec;25(12):5920-5932. doi: 10.1109/TIP.2016.2615424. Epub 2016 Oct 5.
Facial expression is a temporally dynamic event which can be decomposed into a set of muscle motions occurring in different facial regions over various time intervals. For dynamic expression recognition, two key issues, temporal alignment and semantics-aware dynamic representation, must be taken into account. In this paper, we attempt to solve both problems via manifold modeling of videos based on a novel mid-level representation, i.e., expressionlet. Specifically, our method contains three key stages: 1) each expression video clip is characterized as a spatial-temporal manifold (STM) formed by dense low-level features; 2) a universal manifold model (UMM) is learned over all low-level features and represented as a set of local modes to statistically unify all the STMs; and 3) the local modes on each STM can be instantiated by fitting to the UMM, and the corresponding expressionlet is constructed by modeling the variations in each local mode. With the above strategy, expression videos are naturally aligned both spatially and temporally. To enhance the discriminative power, the expressionlet-based STM representation is further processed with discriminant embedding. Our method is evaluated on four public expression databases, CK+, MMI, Oulu-CASIA, and FERA. In all cases, our method outperforms the known state of the art by a large margin.
面部表情是一个随时间动态变化的事件，它可以分解为在不同面部区域、不同时间间隔内发生的一组肌肉运动。对于动态表情识别，必须考虑两个关键问题：时间对齐和语义感知动态表示。在本文中，我们尝试通过基于一种新颖的中级表示（即表情基元）对视频进行流形建模来解决这两个问题。具体来说，我们的方法包含三个关键阶段：1）每个表情视频片段被表征为一个由密集低级特征形成的时空流形（STM）；2）在所有低级特征上学习一个通用流形模型（UMM），并将其表示为一组局部模式，以统计方式统一所有STM；3）每个STM上的局部模式可以通过拟合UMM来实例化，并且通过对每个局部模式中的变化进行建模来构建相应的表情基元。通过上述策略，表情视频在空间和时间上自然对齐。为了增强判别能力，基于表情基元的STM表示通过判别嵌入进一步处理。我们的方法在四个公开的表情数据库CK+、MMI、奥卢-中科院自动化所（Oulu-CASIA）和FERA上进行了评估。在所有情况下，我们的方法都大幅超越了已知的现有技术水平。
