Université de technologie de Compiègne, CNRS, Biomechanics and Bioengineering, Centre de Recherche Royallieu, CS 60 319 - 60 203, Compiègne Cedex, France.
Univ. Lille, CNRS, Centrale Lille, UMR 9013 - LaMcube - Laboratoire de Mécanique, Multiphysique, Multiéchelle, 59655 Villeneuve d'Ascq Cedex, F-59000, Lille, France.
Med Biol Eng Comput. 2021 Jun;59(6):1235-1244. doi: 10.1007/s11517-021-02383-1. Epub 2021 May 24.
Facial expression recognition plays an essential role in human conversation and human-computer interaction. Previous research studies have recognized facial expressions mainly based on 2D image processing requiring sensitive feature engineering and conventional machine learning approaches. The purpose of the present study was to recognize facial expressions by applying a new class of deep learning called geometric deep learning directly on 3D point cloud data. Two databases (Bosphorus and SIAT-3DFE) were used. The Bosphorus database includes sixty-five subjects with seven basic expressions (i.e., anger, disgust, fearness, happiness, sadness, surprise, and neutral). The SIAT-3DFE database has 150 subjects and 4 basic facial expressions (neutral, happiness, sadness, and surprise). First, preprocessing procedures such as face center cropping, data augmentation, and point cloud denoising were applied on 3D face scans. Then, a geometric deep learning model called PointNet++ was applied. A hyperparameter tuning process was performed to find the optimal model parameters. Finally, the developed model was evaluated using the recognition rate and confusion matrix. The facial expression recognition accuracy on the Bosphorus database was 69.01% for 7 expressions and could reach 85.85% when recognizing five specific expressions (anger, disgust, happiness, surprise, and neutral). The recognition rate was 78.70% with the SIAT-3DFE database. The present study suggested that 3D point cloud could be directly processed for facial expression recognition by using geometric deep learning approach. In perspectives, the developed model will be applied for facial palsy patients to guide and optimize the functional rehabilitation program.
面部表情识别在人类对话和人机交互中起着至关重要的作用。先前的研究主要基于 2D 图像处理来识别面部表情，这需要敏感的特征工程和传统的机器学习方法。本研究旨在通过直接在 3D 点云数据上应用一类新的深度学习方法——几何深度学习来识别面部表情。使用了两个数据库（博斯普鲁斯和 SIAT-3DFE）。博斯普鲁斯数据库包括六十五名受试者的七种基本表情（愤怒、厌恶、恐惧、快乐、悲伤、惊讶和中性）。SIAT-3DFE 数据库有 150 名受试者和四种基本面部表情（中性、快乐、悲伤和惊讶）。首先，对面部扫描进行了预处理，包括面部中心裁剪、数据增强和点云去噪。然后，应用了一种名为 PointNet++ 的几何深度学习模型。进行了超参数调整过程以找到最佳模型参数。最后，使用识别率和混淆矩阵评估开发的模型。在博斯普鲁斯数据库中，七种表情的识别准确率为 69.01%，当识别五种特定表情（愤怒、厌恶、快乐、惊讶和中性）时，识别准确率可达 85.85%。在 SIAT-3DFE 数据库中的识别率为 78.70%。本研究表明，可以通过使用几何深度学习方法直接处理 3D 点云进行面部表情识别。从观点来看，开发的模型将应用于面瘫患者，以指导和优化功能康复计划。
