Cheese Eleanor, Bichoo Raouef Ahmed, Grover Kartikae, Dumitru Dorin, Zenonos Alexandros, Groark Joanne, Gibson Douglas, Pope Rebecca
Roche Products Ltd UK, Welwyn Garden City, United Kingdom.
Hull University Teaching Hospital NHS Trust, Hull, United Kingdom.
J Med Internet Res. 2025 Jun 20;27:e70971. doi: 10.2196/70971.
Having well-informed patients is crucial to enhancing patient satisfaction, quality of life, and health outcomes, which in turn optimizes health care use. Traditional methods of delivering information, such as booklets and leaflets, are often ineffective and can overwhelm patients. Educational videos represent a promising alternative; however, their production typically requires significant time and financial resources. Video production using generative artificial intelligence (AI) technology may provide a solution to this problem.
This study aimed to use natural language processing (NLP) to understand free-text patient feedback on 1 of 7 AI-generated patient educational videos created in collaboration with Roche UK and the Hull University Teaching Hospitals NHS Trust breast cancer team, titled "Breast Cancer Follow Up Programme."
A survey was sent to 400 patients who had completed the breast cancer treatment pathway, and 98 (24.5%) free-text responses were received for the question "Any comments or suggestions to improve its [the video's] contents?" We applied and evaluated different NLP machine learning techniques to draw insights from these unstructured data, namely sentiment analysis, topic modeling, summarization, and term frequency-inverse document frequency word clouds.
Sentiment analysis showed that 81% (79/98) of the responses were positive or neutral, while negative comments were predominantly related to the AI avatar. Topic modeling using BERTopic with k-means clustering was found to be the most effective model and identified 4 key topics: the breast cancer treatment pathway, video content, the digital avatar or narrator, and short responses with little or no content. The term frequency-inverse document frequency word clouds indicated positive sentiment about the treatment pathway (eg, "reassured" and "faultless") and video content (eg, "informative" and "clear"), whereas the AI avatar was often described negatively (eg, "impersonal"). Summarization using the text-to-text transfer transformer model effectively created summaries of the responses by topic.
This study demonstrates the success of NLP techniques in efficiently generating insights into patient feedback related to generative AI educational content. Combining NLP methods resulted in clear visuals and insights, enhancing the understanding of patient feedback. Analysis of free-text responses provided clinicians at Hull University Teaching Hospitals NHS Trust with deeper insights than those obtained from quantitative Likert scale responses alone. Importantly, the results validate the use of generative AI in creating patient educational videos, highlighting its potential to address the challenges of costly video production and the limitations of traditional, often overwhelming educational leaflets. Despite the positive overall feedback, negative comments focused on the technical aspects of the AI avatar, indicating areas for improvement. We advocate that patients who receive AI avatar explanations are counseled that this technology is intended to supplement, not replace, human health care interactions. Future investigations are needed to confirm the ongoing effectiveness of these educational tools.
让患者充分了解信息对于提高患者满意度、生活质量和健康结果至关重要，这反过来又能优化医疗保健的使用。传统的信息传递方式，如手册和传单，往往效果不佳，还可能使患者应接不暇。教育视频是一种很有前景的替代方式；然而，其制作通常需要大量的时间和资金。使用生成式人工智能（AI）技术制作视频可能为解决这一问题提供方案。
本研究旨在使用自然语言处理（NLP）来理解患者对与英国罗氏公司和赫尔大学教学医院国民保健服务信托基金乳腺癌团队合作制作的7部人工智能生成的患者教育视频之一“乳腺癌随访计划”的自由文本反馈。
向400名完成乳腺癌治疗流程的患者发送了一份调查问卷，对于“对改进其[视频]内容有任何意见或建议吗？”这一问题，收到了98份（24.5%）自由文本回复。我们应用并评估了不同的NLP机器学习技术，以便从这些非结构化数据中得出见解，即情感分析、主题建模、摘要生成和词频逆文档频率词云。
情感分析显示，81%（79/98）的回复是积极或中性的，而负面评论主要与人工智能虚拟形象有关。使用带有k均值聚类的BERTopic进行主题建模被发现是最有效的模型，并确定了4个关键主题：乳腺癌治疗流程、视频内容、数字虚拟形象或旁白以及内容很少或没有内容的简短回复。词频逆文档频率词云表明对治疗流程（如“放心”和“完美无缺”）和视频内容（如“信息丰富”和“清晰”）有积极的情感，而人工智能虚拟形象经常被负面描述（如“缺乏人情味”）。使用文本到文本迁移变换器模型进行摘要生成有效地按主题创建了回复的摘要。
本研究证明了NLP技术在有效生成与生成式人工智能教育内容相关的患者反馈见解方面的成功。结合NLP方法产生了清晰的可视化结果和见解，增强了对患者反馈的理解。对自由文本回复的分析为赫尔大学教学医院国民保健服务信托基金的临床医生提供了比仅从定量李克特量表回复中获得的更深入的见解。重要的是，结果验证了生成式人工智能在创建患者教育视频中的应用，突出了其应对高成本视频制作挑战以及传统的、往往令人应接不暇的教育传单局限性的潜力。尽管总体反馈是积极的，但负面评论集中在人工智能虚拟形象的技术方面，表明存在改进的空间。我们主张，对于接受人工智能虚拟形象解释的患者，应告知他们这项技术旨在补充而非取代人类医疗保健互动。未来需要进行调查以确认这些教育工具的持续有效性。
