Section of Emergency Medicine, Children's Hospital Colorado, University of Colorado School of Medicine, Aurora, Colorado, USA.
Division of Emergency Medicine, Department of Medicine, McMaster University, Hamilton, ON, Canada.
Perspect Med Educ. 2020 Oct;9(5):302-306. doi: 10.1007/s40037-020-00589-x.
Podcasts are increasingly being used for medical education. Studies have found that the assessment of the quality of online resources can be challenging. We sought to determine the reliability of gestalt quality assessment of education podcasts in emergency medicine.
An international, interprofessional sample of raters was recruited through social media, direct contact, and the extended personal network of the study team. Each participant listened to eight podcasts (selected to include a variety of accents, number of speakers, and topics) and rated the quality of that podcast on a seven-point Likert scale. Phi coefficients were calculated within each group and overall. Decision studies were conducted using a phi of 0.8.
A total of 240 collaborators completed all eight surveys and were included in the analysis. Attendings, medical students, and physician assistants had the lowest individual-level variance and thus the lowest number of required raters to reliably evaluate quality (phi >0.80). Overall, 20 raters were required to reliably evaluate the quality of emergency medicine podcasts.
Gestalt ratings of quality from approximately 20 health professionals are required to reliably assess the quality of a podcast. This finding should inform future work focused on developing and validating tools to support the evaluation of quality in these resources.
播客越来越多地被用于医学教育。研究发现，评估在线资源的质量具有挑战性。我们旨在确定急诊医学教育播客整体质量评估的可靠性。
通过社交媒体、直接联系和研究团队的扩展个人网络，招募了国际跨专业的评估者样本。每位参与者收听了 8 个播客（选择包括各种口音、演讲者人数和主题），并对该播客的质量进行了 7 分李克特量表评分。在每个组和总体内计算了 phi 系数。使用 phi 值为 0.8 进行决策研究。
共有 240 名合作者完成了所有八项调查并纳入分析。主治医生、医学生和医师助理的个体差异最小，因此可靠评估质量所需的评估者人数最少（phi >0.80）。总体而言，需要 20 名评估者来可靠评估急诊医学播客的质量。
需要大约 20 名卫生专业人员的整体质量评分才能可靠评估播客的质量。这一发现应该为未来专注于开发和验证工具以支持这些资源质量评估的工作提供信息。
