State Key Laboratory of Tribology, Department of Mechanical Engineering, Tsinghua University, Beijing, 100084, China.
National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China.
Nat Commun. 2022 Mar 17;13(1):1401. doi: 10.1038/s41467-022-29083-0.
Lip language is an effective method of voice-off communication in daily life for people with vocal cord lesions and laryngeal and lingual injuries without occupying the hands. Collection and interpretation of lip language is challenging. Here, we propose the concept of a novel lip-language decoding system with self-powered, low-cost, contact and flexible triboelectric sensors and a well-trained dilated recurrent neural network model based on prototype learning. The structural principle and electrical properties of the flexible sensors are measured and analysed. Lip motions for selected vowels, words, phrases, silent speech and voice speech are collected and compared. The prototype learning model reaches a test accuracy of 94.5% in training 20 classes with 100 samples each. The applications, such as identity recognition to unlock a gate, directional control of a toy car and lip-motion to speech conversion, work well and demonstrate great feasibility and potential. Our work presents a promising way to help people lacking a voice live a convenient life with barrier-free communication and boost their happiness, enriches the diversity of lip-language translation systems and will have potential value in many applications.
唇语是声带病变和喉舌损伤患者在不占用双手的情况下进行声音交流的有效方法。唇语的采集和解读具有一定的挑战性。在这里，我们提出了一种新型的唇语解码系统的概念，该系统使用自供电、低成本、接触式和灵活的摩擦电传感器，以及基于原型学习的训练有素的扩张循环神经网络模型。我们测量和分析了柔性传感器的结构原理和电气性能。采集并比较了选定的元音、单词、短语、无声言语和言语唇动。原型学习模型在 20 个类别中每个类别训练 100 个样本时，测试准确率达到 94.5%。身份识别解锁门、玩具车的方向控制以及唇动到语音的转换等应用效果良好，展示了很大的可行性和潜力。我们的工作为那些没有声音的人提供了一种有希望的方式，帮助他们实现无障碍沟通，过上便利的生活，提高幸福感，丰富了唇语翻译系统的多样性，在许多应用中具有潜在价值。
