School of Public Health-Bloomington, Indiana University, Bloomington, IN, United States.
J Med Internet Res. 2020 Aug 3;22(8):e19018. doi: 10.2196/19018.
Almost half (46%) of Americans have used a smart assistant of some kind (eg, Apple Siri), and 25% have used a stand-alone smart assistant (eg, Amazon Echo). This positions smart assistants as potentially useful modalities for retrieving health-related information; however, the accuracy of smart assistant responses lacks rigorous evaluation.
This study aimed to evaluate the levels of accuracy, misinformation, and sentiment in smart assistant responses to human papillomavirus (HPV) vaccination-related questions.
We systematically examined responses to questions about the HPV vaccine from the following four most popular smart assistants: Apple Siri, Google Assistant, Amazon Alexa, and Microsoft Cortana. One team member posed 10 questions to each smart assistant and recorded all queries and responses. Two raters independently coded all responses (κ=0.85). We then assessed differences among the smart assistants in terms of response accuracy, presence of misinformation, and sentiment regarding the HPV vaccine.
A total of 103 responses were obtained from the 10 questions posed across the smart assistants. Google Assistant data were excluded owing to nonresponse. Over half (n=63, 61%) of the responses of the remaining three smart assistants were accurate. We found statistically significant differences across the smart assistants (N=103, χ=7.807, P=.02), with Cortana yielding the greatest proportion of misinformation. Siri yielded the greatest proportion of accurate responses (n=26, 72%), whereas Cortana yielded the lowest proportion of accurate responses (n=33, 54%). Most response sentiments across smart assistants were positive (n=65, 64%) or neutral (n=18, 18%), but Cortana's responses yielded the largest proportion of negative sentiment (n=7, 12%).
Smart assistants appear to be average-quality sources for HPV vaccination information, with Alexa responding most reliably. Cortana returned the largest proportion of inaccurate responses, the most misinformation, and the greatest proportion of results with negative sentiments. More collaboration between technology companies and public health entities is necessary to improve the retrieval of accurate health information via smart assistants.
近一半（46%）的美国人使用过某种智能助手（如苹果的 Siri），25%的人使用过独立的智能助手（如亚马逊的 Echo）。这使得智能助手成为检索与健康相关信息的潜在有用方式；然而，智能助手的响应准确性缺乏严格的评估。
本研究旨在评估智能助手对人乳头瘤病毒（HPV）疫苗接种相关问题的响应准确性、错误信息和情绪。
我们系统地检查了来自以下四种最受欢迎的智能助手的关于 HPV 疫苗的问题的响应：苹果的 Siri、谷歌的 Assistant、亚马逊的 Alexa 和微软的 Cortana。一名团队成员向每个智能助手提出了 10 个问题，并记录了所有的查询和响应。两名评分者独立对所有的响应进行编码（κ=0.85）。然后，我们评估了在响应准确性、错误信息存在以及 HPV 疫苗相关情绪方面，不同智能助手之间的差异。
在智能助手之间提出的 10 个问题共获得了 103 个响应。由于没有响应，谷歌助手的数据被排除在外。在其余三个智能助手的 63 个（61%）响应中，超过一半是准确的。我们发现，在智能助手之间存在统计学显著差异（N=103，χ=7.807，P=.02），Cortana 产生的错误信息比例最大。Siri 产生的准确响应比例最大（n=26，72%），而 Cortana 产生的准确响应比例最小（n=33，54%）。大多数智能助手的响应情绪是积极的（n=65，64%）或中立的（n=18，18%），但 Cortana 的响应产生了最大比例的消极情绪（n=7，12%）。
智能助手似乎是 HPV 疫苗接种信息的中等质量来源，Alexa 的响应最可靠。Cortana 返回的不准确响应、错误信息和带有负面情绪的结果比例最大。科技公司和公共卫生实体之间需要更多的合作，才能通过智能助手更准确地检索健康信息。
