Faculty of Engineering Technology, Hung Vuong University, Viet Tri City 35100, Vietnam.
Department of Intelligent Computer Systems, Czestochowa University of Technology, 42-218 Czestochowa, Poland.
Sensors (Basel). 2023 May 27;23(11):5121. doi: 10.3390/s23115121.
Human activity recognition (HAR) is an important research problem in computer vision. This problem is widely applied to building applications in human-machine interactions, monitoring, etc. Especially, HAR based on the human skeleton creates intuitive applications. Therefore, determining the current results of these studies is very important in selecting solutions and developing commercial products. In this paper, we perform a full survey on using deep learning to recognize human activity based on three-dimensional (3D) human skeleton data as input. Our research is based on four types of deep learning networks for activity recognition based on extracted feature vectors: Recurrent Neural Network (RNN) using extracted activity sequence features; Convolutional Neural Network (CNN) uses feature vectors extracted based on the projection of the skeleton into the image space; Graph Convolution Network (GCN) uses features extracted from the skeleton graph and the temporal-spatial function of the skeleton; Hybrid Deep Neural Network (Hybrid-DNN) uses many other types of features in combination. Our survey research is fully implemented from models, databases, metrics, and results from 2019 to March 2023, and they are presented in ascending order of time. In particular, we also carried out a comparative study on HAR based on a 3D human skeleton on the KLHA3D 102 and KLYOGA3D datasets. At the same time, we performed analysis and discussed the obtained results when applying CNN-based, GCN-based, and Hybrid-DNN-based deep learning networks.
人体活动识别（HAR）是计算机视觉中的一个重要研究问题。该问题广泛应用于人机交互、监控等应用程序的构建。特别是，基于人体骨骼的 HAR 可以创建直观的应用程序。因此，在选择解决方案和开发商业产品时，确定这些研究的当前结果非常重要。在本文中，我们对使用深度学习基于三维（3D）人体骨骼数据作为输入来识别人体活动进行了全面调查。我们的研究基于四种用于基于提取特征向量的活动识别的深度学习网络：使用提取的活动序列特征的递归神经网络（RNN）；基于骨骼投影到图像空间的特征向量的卷积神经网络（CNN）；使用从骨骼图和骨骼的时空函数中提取的特征的图卷积网络（GCN）；混合深度神经网络（Hybrid-DNN），结合使用许多其他类型的特征。我们的调查研究完全是基于 2019 年 3 月至 2023 年的模型、数据库、指标和结果实现的，并按照时间顺序递增呈现。特别是，我们还在 KLHA3D102 和 KLYOGA3D 数据集上对基于 3D 人体骨骼的 HAR 进行了比较研究。同时，我们对基于 CNN、GCN 和 Hybrid-DNN 的深度学习网络的应用进行了分析和讨论。
