Park Yoon Soo, Zar Fred A, Norcini John J, Tekian Ara
a Department of Medical Education , University of Illinois at Chicago College of Medicine , Chicago , Illinois , USA.
b Department of Medicine , University of Illinois at Chicago College of Medicine , Chicago , Illinois , USA.
Teach Learn Med. 2016;28(2):135-45. doi: 10.1080/10401334.2016.1146607.
CONSTRUCT: This study examines validity evidence of end-of-rotation evaluation scores used to measure competencies and milestones as part of the Next Accreditation System (NAS) of the Accreditation Council for Graduate Medical Education (ACGME).
Since the implementation of the milestones, end-of-rotation evaluations have surfaced as a potentially useful assessment method. However, validity evidence on the use of rotation evaluation scores as part of the NAS has not been studied. This article examines validity evidence for end-of-rotation evaluations that can contribute to developing guidelines that support the NAS.
Data from 2,701 end-of-rotation evaluations measuring 21 out of 22 Internal Medicine milestones for 142 residents were analyzed (July 2013-June 2014). Descriptive statistics were used to measure the distribution of ratings by evaluators (faculty, n = 116; fellows, n = 59; peer-residents, n = 131), by postgraduate years. Generalizability analysis and higher order confirmatory factor analysis were used to examine the internal structure of ratings. Psychometric implications for combining evaluation scores using composite score reliability were examined.
Milestone ratings were significantly higher for each subsequent year of training (15/21 milestones). Faculty evaluators had greater variability in ratings across milestones, compared to fellows and residents; faculty ratings were generally correlated with milestone ratings from fellows (r = .45) and residents (r = .25), but lower correlations were found for Professionalism and Interpersonal and Communication Skills. The Φ-coefficient was .71, indicating good reliability. Internal structure supported a 6-factor solution, corresponding to the hierarchical relationship between the milestones and the 6 core competencies. Evaluation scores corresponding to Patient Care, Medical Knowledge, and Practice-Based Learning and Improvement had higher correlations to milestones reported to the ACGME. Mean evaluation ratings predicted problem residents (odds ratio = 5.82, p < .001).
Guidelines for rotation evaluations proposed in this study provide useful solutions that can help program directors make decisions on resident progress and contribute to assessment systems in graduate medical education.
构建：本研究检验了作为研究生医学教育认证委员会（ACGME）下一代认证系统（NAS）一部分的轮转结束评估分数用于衡量能力和里程碑的效度证据。
自里程碑实施以来，轮转结束评估已成为一种潜在有用的评估方法。然而，关于将轮转评估分数用作NAS一部分的效度证据尚未得到研究。本文检验了轮转结束评估的效度证据，这有助于制定支持NAS的指南。
分析了2013年7月至2014年6月期间对142名住院医师进行的22项内科里程碑中21项的2701份轮转结束评估数据。描述性统计用于衡量评估者（教员，n = 116；研究员，n = 59；住院医师同伴，n = 131）按研究生年级划分的评分分布。采用概化分析和高阶验证性因素分析来检验评分的内部结构。研究了使用综合评分信度组合评估分数的心理测量学意义。
在随后的每年培训中，里程碑评分显著更高（21项里程碑中的15项）。与研究员和住院医师相比，教员评估者在各里程碑的评分上具有更大的变异性；教员评分通常与研究员（r = 0.45）和住院医师（r = 0.25）的里程碑评分相关，但在职业素养以及人际和沟通技能方面的相关性较低。Φ系数为0.71，表明信度良好。内部结构支持一个6因素解决方案，对应于里程碑与6项核心能力之间的层次关系。与向ACGME报告的里程碑相关度更高的是对应患者照护、医学知识以及基于实践的学习与改进的评估分数。平均评估评分可预测问题住院医师（优势比 = 5.82，p < 0.001）。
本研究中提出的轮转评估指南提供了有用的解决方案，可帮助项目主任就住院医师的进展做出决策，并有助于研究生医学教育的评估系统。
