Isaak Robert S, Chen Fei, Martinelli Susan M, Arora Harendra, Zvara David A, Hobbs Gene, Stiegler Marjorie P
From the Department of Anesthesiology (R.S.I., S.M.M., F.C., H.A., D.A.Z., M.P.S.), University of North Carolina at Chapel Hill School of Medicine, and School of Medicine and Department of Pediatrics (G.H.), University of North Carolina at Chapel Hill, Chapel Hill, NC.
Simul Healthc. 2018 Jun;13(3):201-210. doi: 10.1097/SIH.0000000000000285.
The Accreditation Council for Graduate Medical Education requires biannual evaluation of anesthesiology residents on 25 subcompetency milestones. Some milestone domains are particularly challenging to repeatedly and reliably observe during clinical care. Simulation-Based Milestones Assessment (SBMA) may help overcome these challenges. However, few studies have examined the external validation of simulation assessment scores (ie, the relationships between simulation-based assessment scores and other standard measures of ability) for milestones. This study analyzed whether SBMA scores (1) discriminate by postgraduate year, (2) improve over time, and (3) correlate with traditional measures of performance.
This is a retrospective analysis of 55 residents' SBMA data from 30 scenarios for two academic years. Each scenario was evaluated for time-in-training discrimination. Scenarios were then analyzed for SBMA scoring trends over time, and SBMA scores were compared with residents' clinical evaluations.
Twenty-four SBMA scenarios discriminated by postgraduate year. Repeated measure analysis of variance showed statistically significant between-session score improvements (F (3, 54) = 17.79, P < 0.001). Pearson correlation coefficients demonstrated moderate to strong correlation between SBMA and clinical evaluations: January 2015 r = 0.67, P < 0.01 (n = 27); May 2015 r = 0.43, P = 0.09 (n = 17); November 2015 r = 0.70, P < 0.01 (n = 24); and April 2016 r = 70, P < 0.01 (n = 27).
The associations between SBMA scores and experience level, time-in-training, and clinical performance evaluations provide evidence that SBMA may be used as metrics of residents' Accreditation Council for Graduate Medical Education milestone competencies.
毕业后医学教育认证委员会要求每半年对麻醉学住院医师在25个亚能力里程碑方面进行评估。在临床护理过程中，反复且可靠地观察某些里程碑领域极具挑战性。基于模拟的里程碑评估（SBMA）或许有助于克服这些挑战。然而，很少有研究检验模拟评估分数（即基于模拟的评估分数与其他能力标准衡量指标之间的关系）对里程碑的外部效度。本研究分析了SBMA分数是否（1）按研究生年级进行区分，（2）随时间推移有所提高，以及（3）与传统绩效衡量指标相关。
这是一项对两个学年中55名住院医师来自30个场景的SBMA数据的回顾性分析。对每个场景进行培训时间区分度评估。然后分析各场景随时间的SBMA评分趋势，并将SBMA分数与住院医师的临床评估进行比较。
24个SBMA场景按研究生年级进行了区分。重复测量方差分析显示各场次之间的分数有统计学意义的提高（F(3, 54) = 17.79，P < 0.001）。Pearson相关系数表明SBMA与临床评估之间存在中度至高度相关性：2015年1月r = 0.67，P < 0.01（n = 27）；2015年5月r = 0.43，P = 0.09（n = 17）；2015年11月r = 0.70，P < 0.01（n = 24）；以及2016年4月r = 70，P < 0.01（n = 27）。
SBMA分数与经验水平、培训时间和临床表现评估之间的关联提供了证据，表明SBMA可用作住院医师毕业后医学教育认证委员会里程碑能力的衡量指标。
