Chan Teresa M, Sherbino Jonathan, Mercuri Mathew
J Grad Med Educ. 2017 Dec;9(6):724-729. doi: 10.4300/JGME-D-17-00086.1.
Competency-based medical education requires frequent assessment to tailor learning experiences to the needs of trainees. In 2012, we implemented the McMaster Modular Assessment Program, which captures shift-based assessments of resident global performance.
We described patterns (ie, trends and sources of variance) in aggregated workplace-based assessment data.
Emergency medicine residents and faculty members from 3 Canadian university-affiliated, urban, tertiary care teaching hospitals participated in this study. During each shift, supervising physicians rated residents' performance using a behaviorally anchored scale that hinged on endorsements for progression. We used a multilevel regression model to examine the relationship between global rating scores and time, adjusting for data clustering by resident and rater.
We analyzed data from 23 second-year residents between July 2012 and June 2015, which yielded 1498 unique ratings (65 ± 18.5 per resident) from 82 raters. The model estimated an average score of 5.7 ± 0.6 at baseline, with an increase of 0.005 ± 0.01 for each additional assessment. There was significant variation among residents' starting score (y-intercept) and trajectory (slope).
Our model suggests that residents begin at different points and progress at different rates. Meta-raters such as program directors and Clinical Competency Committee members should bear in mind that progression may take time and learning trajectories will be nuanced. Individuals involved in ratings should be aware of sources of noise in the system, including the raters themselves.
基于能力的医学教育要求频繁进行评估，以便根据学员的需求调整学习体验。2012年，我们实施了麦克马斯特模块化评估计划，该计划收集基于轮班的住院医师整体表现评估。
我们描述了汇总的基于工作场所的评估数据中的模式（即趋势和方差来源）。
来自加拿大3所大学附属的城市三级护理教学医院的急诊医学住院医师和教员参与了本研究。在每个轮班期间，监督医师使用基于行为的量表对住院医师的表现进行评分，该量表取决于对进步的认可。我们使用多层次回归模型来检验整体评分与时间之间的关系，并对住院医师和评分者的数据聚类进行调整。
我们分析了2012年7月至2015年6月期间23名二年级住院医师的数据，这些数据产生了来自82名评分者的1498个独特评分（每位住院医师65±18.5个）。该模型估计基线时的平均得分为5.7±0.6，每次额外评估增加0.005±0.01。住院医师的起始分数（y轴截距）和轨迹（斜率）存在显著差异。
我们的模型表明，住院医师起点不同，进步速度也不同。项目主任和临床能力委员会成员等元评分者应记住，进步可能需要时间，学习轨迹也会有细微差别。参与评分的个人应意识到系统中的噪声来源，包括评分者自身。
