J.R. Kogan is associate dean, Student Success and Professional Development, and professor of medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania; ORCID: https://orcid.org/0000-0001-8426-9506 .
C.J. Dine is associate dean, Evaluation and Assessment, and associate professor of medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania; ORCID: https://orcid.org/0000-0001-5894-0861 .
Acad Med. 2023 Feb 1;98(2):237-247. doi: 10.1097/ACM.0000000000004819. Epub 2022 Jul 21.
Prior research evaluating workplace-based assessment (WBA) rater training effectiveness has not measured improvement in narrative comment quality and accuracy, nor accuracy of prospective entrustment-supervision ratings. The purpose of this study was to determine whether rater training, using performance dimension and frame of reference training, could improve WBA narrative comment quality and accuracy. A secondary aim was to assess impact on entrustment rating accuracy.
This single-blind, multi-institution, randomized controlled trial of a multifaceted, longitudinal rater training intervention consisted of in-person training followed by asynchronous online spaced learning. In 2018, investigators randomized 94 internal medicine and family medicine physicians involved with resident education. Participants assessed 10 scripted standardized resident-patient videos at baseline and follow-up. Differences in holistic assessment of narrative comment accuracy and specificity, accuracy of individual scenario observations, and entrustment rating accuracy were evaluated with t tests. Linear regression assessed impact of participant demographics and baseline performance.
Seventy-seven participants completed the study. At follow-up, the intervention group (n = 41), compared with the control group (n = 36), had higher scores for narrative holistic specificity (2.76 vs 2.31, P < .001, Cohen V = .25), accuracy (2.37 vs 2.06, P < .001, Cohen V = .20) and mean quantity of accurate (6.14 vs 4.33, P < .001), inaccurate (3.53 vs 2.41, P < .001), and overall observations (2.61 vs 1.92, P = .002, Cohen V = .47). In aggregate, the intervention group had more accurate entrustment ratings (58.1% vs 49.7%, P = .006, Phi = .30). Baseline performance was significantly associated with performance on final assessments.
Quality and specificity of narrative comments improved with rater training; the effect was mitigated by inappropriate stringency. Training improved accuracy of prospective entrustment-supervision ratings, but the effect was more limited. Participants with lower baseline rating skill may benefit most from training.
先前评估基于工作场所的评估（WBA）评估者培训效果的研究并未衡量叙述性评论质量和准确性的提高，也未衡量前瞻性委托-监督评分的准确性。本研究的目的是确定使用绩效维度和参考框架培训是否可以提高 WBA 叙述性评论的质量和准确性。次要目的是评估对委托评分准确性的影响。
这是一项单盲、多机构、随机对照试验，采用多方面、纵向的评估者培训干预措施，包括面对面培训和异步在线间隔学习。2018 年，研究人员对参与住院医师教育的 94 名内科和家庭医学医师进行了随机分组。参与者在基线和随访时评估了 10 个脚本标准化住院医师-患者视频。使用 t 检验评估整体评估叙述性评论准确性和特异性、个别情景观察准确性以及委托评分准确性的差异。线性回归评估参与者人口统计学特征和基线表现的影响。
77 名参与者完成了研究。在随访时，干预组（n=41）与对照组（n=36）相比，叙述性整体特异性评分更高（2.76 对 2.31，P<.001，Cohen V=0.25），准确性评分更高（2.37 对 2.06，P<.001，Cohen V=0.20），准确观察的平均值更高（6.14 对 4.33，P<.001），不准确观察的平均值更高（3.53 对 2.41，P<.001），以及整体观察的平均值更高（2.61 对 1.92，P=.002，Cohen V=0.47）。总体而言，干预组的委托评分更准确（58.1% 对 49.7%，P=.006，Phi=0.30）。基线表现与最终评估表现显著相关。
评估者培训提高了叙述性评论的质量和特异性；但不当的严格性降低了效果。培训提高了前瞻性委托-监督评分的准确性，但效果更为有限。基线评分技能较低的参与者可能从培训中获益最多。
