State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, China.
Key Laboratory of Acoustic Visual Technology and Intelligent Control System, Ministry of Culture and Tourism, Communication University of China, Beijing 100024, China.
Sensors (Basel). 2022 Aug 5;22(15):5855. doi: 10.3390/s22155855.
Gesture recognition based on wearable devices is one of the vital components of human-computer interaction systems. Compared with skeleton-based recognition in computer vision, gesture recognition using wearable sensors has attracted wide attention for its robustness and convenience. Recently, many studies have proposed deep learning methods based on surface electromyography (sEMG) signals for gesture classification; however, most of the existing datasets are built for surface EMG signals, and there is a lack of datasets for multi-category gestures. Due to model limitations and inadequate classification data, the recognition accuracy of these methods cannot satisfy multi-gesture interaction scenarios. In this paper, a multi-category dataset containing 20 gestures is recorded with the help of a wearable device that can acquire surface electromyographic and inertial (IMU) signals. Various two-stream deep learning models are established and improved further. The basic convolutional neural network (CNN), recurrent neural network (RNN), and Transformer models are experimented on with our dataset as the classifier. The CNN and the RNN models' test accuracy is over 95%; however, the Transformer model has a lower test accuracy of 71.68%. After further improvements, the CNN model is introduced into the residual network and augmented to the CNN-Res model, achieving 98.24% accuracy; moreover, it has the shortest training and testing time. Then, after combining the RNN model and the CNN-Res model, the long short term memory (LSTM)-Res model and gate recurrent unit (GRU)-Res model achieve the highest classification accuracy of 99.67% and 99.49%, respectively. Finally, the fusion of the Transformer model and the CNN model enables the Transformer-CNN model to be constructed. Such improvement dramatically boosts the performance of the Transformer module, increasing the recognition accuracy from 71.86% to 98.96%.
基于可穿戴设备的手势识别是人机交互系统的重要组成部分之一。与计算机视觉中的基于骨架的识别相比，使用可穿戴传感器的手势识别因其鲁棒性和便利性而受到广泛关注。最近，许多研究提出了基于表面肌电 (sEMG) 信号的深度学习方法用于手势分类; 然而，现有的大多数数据集都是针对表面 EMG 信号构建的，缺乏多类别手势数据集。由于模型限制和分类数据不足，这些方法的识别精度无法满足多手势交互场景的需求。在本文中，借助可以获取表面肌电和惯性 (IMU) 信号的可穿戴设备，记录了一个包含 20 个手势的多类别数据集。进一步建立和改进了各种双流深度学习模型。将基本卷积神经网络 (CNN)、循环神经网络 (RNN) 和 Transformer 模型作为分类器应用于我们的数据集。CNN 和 RNN 模型的测试准确率均超过 95%; 然而，Transformer 模型的测试准确率较低，为 71.68%。进一步改进后，将 CNN 模型引入到残差网络中，并扩充到 CNN-Res 模型中，准确率达到 98.24%；而且，它具有最短的训练和测试时间。然后，在将 RNN 模型和 CNN-Res 模型相结合后，长短期记忆 (LSTM)-Res 模型和门控循环单元 (GRU)-Res 模型分别实现了最高的分类准确率 99.67%和 99.49%。最后，Transformer 模型和 CNN 模型的融合构建了 Transformer-CNN 模型。这种改进极大地提高了 Transformer 模块的性能，将识别准确率从 71.86%提高到 98.96%。
