is Associate Dean, Student Success and Professional Development, and Professor of Medicine, Perelman School of Medicine, University of Pennsylvania.
is Research Associate for Milestones Evaluation, Accreditation Council for Graduate Medical Education (ACGME).
J Grad Med Educ. 2023 Feb;15(1):81-91. doi: 10.4300/JGME-D-22-00287.1.
Workplace-based assessment (WBA) is a key assessment strategy in competency-based medical education. However, its full potential has not been actualized secondary to concerns with reliability, validity, and accuracy. Frame of reference training (FORT), a rater training technique that helps assessors distinguish between learner performance levels, can improve the accuracy and reliability of WBA, but the effect size is variable. Understanding FORT benefits and challenges help improve this rater training technique.
To explore faculty's perceptions of the benefits and challenges associated with FORT.
Subjects were internal medicine and family medicine physicians (n=41) who participated in a rater training intervention in 2018 consisting of in-person FORT followed by asynchronous online spaced learning. We assessed participants' perceptions of FORT in post-workshop focus groups and an end-of-study survey. Focus groups and survey free text responses were coded using thematic analysis.
All subjects participated in 1 of 4 focus groups and completed the survey. Four benefits of FORT were identified: (1) opportunity to apply skills frameworks via deliberate practice; (2) demonstration of the importance of certain evidence-based clinical skills; (3) practice that improved the ability to discriminate between resident skill levels; and (4) highlighting the importance of direct observation and the dangers using proxy information in assessment. Challenges included time constraints and task repetitiveness.
Participants believe that FORT training serves multiple purposes, including helping them distinguish between learner skill levels while demonstrating the impact of evidence-based clinical skills and the importance of direct observation.
基于工作场所的评估（WBA）是基于能力的医学教育的关键评估策略。然而，由于对可靠性、有效性和准确性的担忧，其全部潜力尚未实现。参考框架培训（FORT）是一种评分员培训技术，可帮助评估者区分学习者的表现水平，可提高 WBA 的准确性和可靠性，但效果大小是可变的。了解 FORT 的优缺点有助于改进这种评分员培训技术。
探讨教师对 FORT 相关益处和挑战的看法。
参与者为参加 2018 年评分员培训干预措施的内科和家庭医学医师（n=41），该干预措施包括面对面的 FORT 以及异步在线间隔学习。我们在研讨会后的焦点小组和期末研究调查中评估了参与者对 FORT 的看法。使用主题分析对焦点小组和调查的自由文本回复进行编码。
所有参与者都参加了 4 个焦点小组中的 1 个，并完成了调查。确定了 FORT 的四个好处：（1）通过刻意练习应用技能框架的机会；（2）展示某些基于证据的临床技能的重要性；（3）提高区分居民技能水平的能力的实践；（4）强调直接观察的重要性和在评估中使用代理信息的危险。挑战包括时间限制和任务重复性。
参与者认为 FORT 培训有多种用途，包括帮助他们区分学习者的技能水平，同时展示基于证据的临床技能的影响和直接观察的重要性。
