Faculty of Informatics, Gunma University, Kiryu 3768515, Japan.
Graduate School of Engineering, Hokkaido University, Sapporo 0608628, Japan.
Sensors (Basel). 2024 Jan 11;24(2):453. doi: 10.3390/s24020453.
Sign language is designed as a natural communication method to convey messages among the deaf community. In the study of sign language recognition through wearable sensors, the data sources are limited, and the data acquisition process is complex. This research aims to collect an American sign language dataset with a wearable inertial motion capture system and realize the recognition and end-to-end translation of sign language sentences with deep learning models. In this work, a dataset consisting of 300 commonly used sentences is gathered from 3 volunteers. In the design of the recognition network, the model mainly consists of three layers: convolutional neural network, bi-directional long short-term memory, and connectionist temporal classification. The model achieves accuracy rates of 99.07% in word-level evaluation and 97.34% in sentence-level evaluation. In the design of the translation network, the encoder-decoder structured model is mainly based on long short-term memory with global attention. The word error rate of end-to-end translation is 16.63%. The proposed method has the potential to recognize more sign language sentences with reliable inertial data from the device.
手语是专为聋人群体设计的一种自然沟通方式，用于传递信息。在利用可穿戴传感器进行手语识别研究中，数据源有限，数据采集过程复杂。本研究旨在利用可穿戴惯性运动捕捉系统收集美国手语数据集，并利用深度学习模型实现手语句子的识别和端到端翻译。在这项工作中，我们从 3 名志愿者那里收集了一个由 300 个常用句子组成的数据集。在识别网络的设计中，模型主要由卷积神经网络、双向长短时记忆和连接时序分类三个部分组成。在单词级别的评估中，模型的准确率达到了 99.07%，在句子级别的评估中准确率达到了 97.34%。在翻译网络的设计中，编码器-解码器结构模型主要基于带有全局注意力机制的长短时记忆。端到端翻译的单词错误率为 16.63%。该方法有望利用设备中可靠的惯性数据识别更多的手语句子。
