Irasiak Anna, Kozak Jan, Piasecki Adam, Stęclik Tomasz
Deartament of Pedagogy, Jan Dlugosz University in Czestochowa, Al. Armii Krajowej 13/15, 42-200 Czestochowa, Poland.
Department of Machine Learning, University of Economics in Katowice, 1 Maja 50, 40-287 Katowice, Poland.
Entropy (Basel). 2023 Jan 6;25(1):120. doi: 10.3390/e25010120.
Automatic translation between the national language and sign language is a complex process similar to translation between two different foreign languages. A very important aspect is the precision of not only manual gestures but also facial expressions, which are extremely important in the overall context of a sentence. In this article, we present the problem of including facial expressions in the automation of Polish-to-Polish Sign Language (PJM) translation-this is part of an ongoing project related to a comprehensive solution allowing for the animation of manual gestures, body movements and facial expressions. Our approach explores the possibility of using action unit (AU) recognition in the automatic annotation of recordings, which in the subsequent steps will be used to train machine learning models. This paper aims to evaluate entropy in real-life translation recordings and analyze the data associated with the detected action units. Our approach has been subjected to evaluation by experts related to Polish Sign Language, and the results obtained allow for the development of further work related to automatic translation into Polish Sign Language.
国家语言和手语之间的自动翻译是一个复杂的过程，类似于两种不同外语之间的翻译。一个非常重要的方面不仅是手势的精确性，还有面部表情的精确性，面部表情在句子的整体语境中极其重要。在本文中，我们提出了在波兰语到波兰手语（PJM）翻译自动化中纳入面部表情的问题——这是一个正在进行的项目的一部分，该项目致力于实现一个全面的解决方案，能够对手势、身体动作和面部表情进行动画处理。我们的方法探索了在录音自动标注中使用动作单元（AU）识别的可能性，在后续步骤中，这些标注将用于训练机器学习模型。本文旨在评估实际翻译录音中的熵，并分析与检测到的动作单元相关的数据。我们的方法已经由波兰手语方面的专家进行了评估，所获得的结果有助于开展与波兰手语自动翻译相关的进一步工作。
