School of Information & Communication Engineering, Beijing Information Science and Technology University, Beijing 100025, China.
Sensors (Basel). 2022 Feb 8;22(3):1270. doi: 10.3390/s22031270.
The 'intention' classification of a user question is an important element of a task-engine driven chatbot. The essence of a user question's intention understanding is the text classification. The transfer learning, such as BERT (Bidirectional Encoder Representations from Transformers) and ERNIE (Enhanced Representation through Knowledge Integration), has put the text classification task into a new level, but the BERT and ERNIE model are difficult to support high QPS (queries per second) intelligent dialogue systems due to computational performance issues. In reality, the simple classification model usually shows a high computational performance, but they are limited by low accuracy. In this paper, we use knowledge of the ERNIE model to distill the FastText model; the ERNIE model works as a teacher model to predict the massive online unlabeled data for data enhancement, and then guides the training of the student model of FastText with better computational efficiency. The FastText model is distilled by the ERNIE model in chatbot intention classification. This not only guarantees the superiority of its original computational performance, but also the intention classification accuracy has been significantly improved.
用户问题的“意图”分类是任务引擎驱动的聊天机器人的重要元素。用户问题意图理解的本质是文本分类。迁移学习，如 BERT（来自转换器的双向编码器表示）和 ERNIE（通过知识集成增强表示），将文本分类任务提升到了一个新的水平，但由于计算性能问题，BERT 和 ERNIE 模型难以支持高 QPS（每秒查询量）智能对话系统。实际上，简单的分类模型通常表现出较高的计算性能，但它们受到准确性较低的限制。在本文中，我们使用 ERNIE 模型的知识来提炼 FastText 模型；ERNIE 模型作为教师模型来预测大规模的在线无标签数据以进行数据增强，然后指导具有更好计算效率的 FastText 学生模型的训练。ERNIE 模型在聊天机器人意图分类中对 FastText 模型进行蒸馏。这不仅保证了其原始计算性能的优越性，而且意图分类的准确性也得到了显著提高。
