Baihan Abdullah, Alutaibi Ahmed I, Alshehri Mohammed, Sharma Sunil Kumar
Computer Science Department, Community College, King Saud University, 11437, Riyadh, Saudi Arabia.
Department of Computer Engineering, College of Computer and Information Sciences, Majmaah University, 11952, Majmaah, Saudi Arabia.
Sci Rep. 2024 Oct 30;14(1):26111. doi: 10.1038/s41598-024-76174-7.
Speech impairment limits a person's capacity for oral and auditory communication. Improvements in communication between the deaf and the general public can be progressed by a real-time sign language detector. Recent studies have contributed to make progress in motion and gesture identification processes using Deep Learning (DL) methods and computer vision. But the development of static and dynamic sign language recognition (SLR) models is still a challenging area of research. The difficulty is in obtaining an appropriate model that addresses the challenges of continuous signs that are independent of the signer. Different signers' speeds, durations, and many other factors make it challenging to create a model with high accuracy and continuity. This study mainly focused on SLR using a modified DL and hybrid optimization approach. Notably, spatial and geometric-based features are extracted via the Visual Geometry Group 16 (VGG16), and motion features are extracted using the optical flow approach. A new DL model, CNNSa-LSTM, is a combination of a Convolutional Neural Network (CNN), Self-Attention (SA), and Long-Short-Term Memory (LSTM) to identify sign language. This model is developed for feature extraction by combining CNNs for spatial analysis with SA mechanisms for focusing on relevant features, while LSTM effectively models temporal dependencies. The proposed CNNSa-LSTM model enhances performance in tasks involving complex, sequential data, such as sign language processing. Besides, a Hybrid Optimizer (HO) is proposed using the Hippopotamus Optimization Algorithm (HOA) and the Pathfinder Algorithm (PFA). The proposed model has been implemented in Python, and it has been evaluated over the existing models in terms of accuracy (98.7%), sensitivity (98.2%), precision (98.5%), Word Error Rate (WER) (0.131), Sign Error Rate (SER) (0.114), and Normalized Discounted Cumulative Gain (NDCG) (98%) as well. The proposed model has recorded the highest accuracy of 98.7%.
言语障碍限制了一个人的口语和听觉交流能力。实时手语检测器可以促进聋人与普通大众之间交流的改善。最近的研究推动了使用深度学习（DL）方法和计算机视觉在运动和手势识别过程方面取得进展。但是静态和动态手语识别（SLR）模型的开发仍然是一个具有挑战性的研究领域。困难在于获得一个合适的模型来应对与手语者无关的连续手语的挑战。不同手语者的速度、时长以及许多其他因素使得创建一个具有高精度和连续性的模型具有挑战性。本研究主要关注使用改进的DL和混合优化方法进行手语识别。值得注意的是，基于空间和几何的特征通过视觉几何组16（VGG16）提取，运动特征使用光流方法提取。一种新的DL模型，CNNSa-LSTM，是卷积神经网络（CNN）、自注意力（SA）和长短期记忆（LSTM）的组合，用于识别手语。该模型通过将用于空间分析的CNN与用于关注相关特征的SA机制相结合来进行特征提取，而LSTM有效地对时间依赖性进行建模。所提出的CNNSa-LSTM模型在涉及复杂序列数据的任务（如手语处理）中提高了性能。此外，还提出了一种使用河马优化算法（HOA）和探路者算法（PFA）的混合优化器（HO）。所提出的模型已在Python中实现，并在准确性（98.7%）、灵敏度（98.2%）、精度（98.5%）、单词错误率（WER）（0.131）、手语错误率（SER）（0.114）和归一化折损累计增益（NDCG）（98%）方面与现有模型进行了评估。所提出的模型记录了最高98.7%的准确率。
