Liu Xiao, Susarla Anjana, Padman Rema
Department of Information Systems, W. P. Carey School of Business, Arizona State University, Tempe, AZ, United States.
Accounting Information Systems, Broad College of Business, Michigan State University, Lansing, MI, United States.
J Med Internet Res. 2025 Apr 8;27:e56080. doi: 10.2196/56080.
An estimated 93% of adults in the United States access the internet, with up to 80% looking for health information. However, only 12% of US adults are proficient enough in health literacy to interpret health information and make informed health care decisions meaningfully. With the vast amount of health information available in multimedia formats on social media platforms such as YouTube and Facebook, there is an urgent need and a unique opportunity to design an automated approach to curate online health information using multiple criteria to meet the health literacy needs of a diverse population.
This study aimed to develop an automated approach to assessing the understandability of patient educational videos according to the Patient Education Materials Assessment Tool (PEMAT) guidelines and evaluating the impact of video understandability on viewer engagement. We also offer insights for content creators and health care organizations on how to improve engagement with these educational videos on user-generated content platforms.
We developed a human-in-the-loop, augmented intelligence approach that explicitly focused on the human-algorithm interaction, combining PEMAT-based patient education constructs mapped to features extracted from the videos, annotations of the videos by domain experts, and cotraining methods from machine learning to assess the understandability of videos on diabetes and classify them. We further examined the impact of understandability on several dimensions of viewer engagement with the videos.
We collected 9873 YouTube videos on diabetes using search keywords extracted from a patient-oriented forum and reviewed by a medical expert. Our machine learning methods achieved a weighted precision of 0.84, a weighted recall of 0.79, and an F-score of 0.81 in classifying video understandability and could effectively identify patient educational videos that medical experts would like to recommend for patients. Videos rated as highly understandable had an average higher view count (average treatment effect [ATE]=2.55; P<.001), like count (ATE=2.95; P<.001), and comment count (ATE=3.10; P<.001) than less understandable videos. In addition, in a user study, 4 medical experts recommended 72% (144/200) of the top 10 videos ranked by understandability compared to 40% (80/200) of the top 10 videos ranked by YouTube's default algorithm for 20 ramdomly selected search keywords.
We developed a human-in-the-loop, scalable algorithm to assess the understandability of health information on YouTube. Our method optimally combines expert input with algorithmic support, enhancing engagement and aiding medical experts in recommending educational content. This solution also guides health care organizations in creating effective patient education materials for underserved health topics.
据估计，美国93%的成年人可以访问互联网，其中高达80%的人在网上查找健康信息。然而，只有12%的美国成年人具备足够的健康素养来解读健康信息并做出明智的医疗保健决策。在YouTube和Facebook等社交媒体平台上，有大量以多媒体形式提供的健康信息，因此迫切需要并存在独特的机会来设计一种自动化方法，使用多种标准来筛选在线健康信息，以满足不同人群的健康素养需求。
本研究旨在开发一种自动化方法，根据患者教育材料评估工具（PEMAT）指南评估患者教育视频的可理解性，并评估视频可理解性对观众参与度的影响。我们还为内容创作者和医疗保健组织提供了关于如何提高在用户生成内容平台上与这些教育视频互动的见解。
我们开发了一种人工参与的增强智能方法，该方法明确关注人机算法交互，将基于PEMAT的患者教育结构映射到从视频中提取的数据特征、领域专家对视频的注释以及机器学习中的协同训练方法，以评估糖尿病视频的可理解性并对其进行分类。我们进一步研究了可理解性对视频观众参与度几个维度的影响。
我们使用从一个面向患者的论坛中提取并经医学专家审核的搜索关键词，在YouTube上收集了9873个关于糖尿病的视频。我们的机器学习方法在对视频可理解性进行分类时，加权精确率达到0.84，加权召回率达到 0.79，F值达到0.81，并且能够有效地识别医学专家希望推荐给患者的患者教育视频。被评为高度可理解的视频，其平均观看次数（平均治疗效果[ATE]=2.55；P<0.001）、点赞数（ATE=2.95；P<0.001）和评论数（ATE= 3.10；P<0.001）均高于可理解性较低的视频。此外，在一项用户研究中，对于20个随机选择的搜索关键词，4位医学专家推荐了按可理解性排名前10的视频中的72%（144/200），而按YouTube默认算法排名前10的视频中只有40%（80/200）被推荐。
我们开发了一种人工参与的、可扩展的算法来评估YouTube上健康信息的可理解性。我们的方法将专家输入与算法支持进行了最佳结合，提高了参与度，并帮助医学专家推荐教育内容。该解决方案还指导医疗保健组织为服务不足的健康主题创建有效的患者教育材料。
