Xiao Wei, Hu Qian, Mohamed Thahir, Gao Zheng, Gao Xibin, Arava Radhika, AbdelHady Mohamed
Amazon Alexa AI, Seattle, WA, United States.
Front Big Data. 2022 Jul 18;5:898050. doi: 10.3389/fdata.2022.898050. eCollection 2022.
Intelligent personal assistants (IPA) enable voice applications that facilitate people's daily tasks. However, due to the complexity and ambiguity of voice requests, some requests may not be handled properly by the standard natural language understanding (NLU) component. In such cases, a simple reply like "Sorry, I don't know" hurts the user's experience and limits the functionality of IPA. In this paper, we propose a two-stage shortlister-reranker recommender system to match third-party voice applications (skills) to unhandled utterances. In this approach, a skill shortlister is proposed to retrieve candidate skills from the skill catalog by calculating both lexical and semantic similarity between skills and user requests. We also illustrate how to build a new system by using observed data collected from a baseline rule-based system, and how the exposure biases can generate discrepancy between offline and human metrics. Lastly, we present two relabeling methods that can handle the incomplete ground truth, and mitigate exposure bias. We demonstrate the effectiveness of our proposed system through extensive offline experiments. Furthermore, we present online A/B testing results that show a significant boost on user experience satisfaction.
智能个人助理（IPA）支持语音应用程序，这些程序可方便人们完成日常任务。然而，由于语音请求的复杂性和模糊性，标准自然语言理解（NLU）组件可能无法正确处理某些请求。在这种情况下，像“对不起，我不知道”这样简单的回复会损害用户体验，并限制IPA的功能。在本文中，我们提出了一种两阶段的候选列表-重排推荐系统，用于将第三方语音应用程序（技能）与未处理的话语进行匹配。在这种方法中，提出了一种技能候选列表，通过计算技能与用户请求之间的词汇和语义相似度，从技能目录中检索候选技能。我们还说明了如何使用从基于基线规则的系统收集的观察数据构建新系统，以及曝光偏差如何在离线指标和人工指标之间产生差异。最后，我们提出了两种重新标记方法，它们可以处理不完整的真实情况，并减轻曝光偏差。我们通过广泛的离线实验证明了我们提出的系统的有效性。此外，我们还展示了在线A/B测试结果，这些结果表明用户体验满意度有显著提高。
