Benko Ľubomír, Munkova Dasa, Munk Michal, Benkova Lucia, Hajek Petr
Constantine the Philosopher University in Nitra, Tr. A. Hlinku 1, 949 01, Nitra, Slovakia.
Science and Research Centre, University of Pardubice, Studentská 84, 532 10, Pardubice, Czech Republic.
Sci Rep. 2024 Apr 23;14(1):9293. doi: 10.1038/s41598-024-59524-3.
The aim of the study is to compare two different approaches to machine translation-statistical and neural-using automatic MT metrics of error rate and residuals. We examined four available online MT systems (statistical Google Translate, neural Google Translate, and two European commission's MT tools-statistical mt@ec and neural eTranslation) through their products (MT outputs). We propose using residual analysis to improve the accuracy of machine translation error rate. Residuals represent a new approach to comparing the quality of statistical and neural MT outputs. The study provides new insights into evaluating machine translation quality from English and German into Slovak through automatic error rate metrics. In the category of prediction and syntactic-semantic correlativeness, statistical MT showed a significantly higher error rate than neural MT. Conversely, in the category of lexical semantics, neural MT showed a significantly higher error rate than statistical MT. The results indicate that relying solely on the reference when determining MT quality is insufficient. However, when combined with residuals, it offers a more objective view of MT quality and facilitates the comparison of statistical MT and neural MT.
本研究的目的是使用错误率和残差等自动机器翻译指标，比较两种不同的机器翻译方法——统计方法和神经方法。我们通过四个可用的在线机器翻译系统（统计型谷歌翻译、神经型谷歌翻译以及欧盟委员会的两个机器翻译工具——统计型mt@ec和神经型eTranslation）的输出结果对它们进行了考察。我们建议使用残差分析来提高机器翻译错误率的准确性。残差代表了一种比较统计型和神经型机器翻译输出质量的新方法。该研究通过自动错误率指标，为评估从英语和德语到斯洛伐克语的机器翻译质量提供了新的见解。在预测和句法-语义相关性类别中，统计型机器翻译的错误率显著高于神经型机器翻译。相反，在词汇语义类别中，神经型机器翻译的错误率显著高于统计型机器翻译。结果表明，在确定机器翻译质量时仅依靠参考译文是不够的。然而，将其与残差相结合，可以更客观地看待机器翻译质量，并便于比较统计型机器翻译和神经型机器翻译。
