M.S. Ryan is associate professor and assistant dean for clinical medical education, Department of Pediatrics, Virginia Commonwealth University, Richmond, Virginia; ORCID: https://orcid.org/0000-0003-3266-9289 .
A. Richards is a graduate student, Department of Biostatistics, Virginia Commonwealth University, Richmond, Virginia.
Acad Med. 2021 Aug 1;96(8):1197-1204. doi: 10.1097/ACM.0000000000003921. Epub 2021 Jan 12.
Assessment of the Core Entrustable Professional Activities for Entering Residency (Core EPAs) requires direct observation of learners in the workplace to support entrustment decisions. The purpose of this study was to examine the internal structure validity evidence of the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) scale when used to assess medical student performance in the Core EPAs across clinical clerkships.
During the 2018-2019 academic year, the Virginia Commonwealth University School of Medicine implemented a mobile-friendly, student-initiated workplace-based assessment (WBA) system to provide formative feedback for the Core EPAs across all clinical clerkships. Students were required to request a specified number of Core EPA assessments in each clerkship. A modified O-SCORE scale (1 = "I had to do" to 4 = "I needed to be in room just in case") was used to rate learner performance. Generalizability theory was applied to assess the generalizability (or reliability) of the assessments. Decision studies were then conducted to determine the number of assessments needed to achieve a reasonable reliability.
A total of 10,680 WBAs were completed on 220 medical students. The majority of ratings were completed on EPA 1 (history and physical) (n = 3,129; 29%) and EPA 6 (oral presentation) (n = 2,830; 26%). Mean scores were similar (3.5-3.6 out of 4) across EPAs. Variance due to the student ranged from 3.5% to 8%, with the majority of the variation due to the rater (29.6%-50.3%) and other unexplained factors. A range of 25 to 63 assessments were required to achieve reasonable reliability (Phi > 0.70).
The O-SCORE demonstrated modest reliability when used across clerkships. These findings highlight specific challenges for implementing WBAs for the Core EPAs including the process for requesting WBAs, rater training, and application of the O-SCORE scale in medical student assessment.
评估住院医师入职的核心可信赖专业活动（Core EPAs）需要直接观察学习者在工作场所的表现，以支持委托决策。本研究的目的是检验渥太华手术能力手术室评估（O-SCORE）量表在评估医学生在各临床实习阶段的核心 EPAs 表现时的内部结构有效性证据。
在 2018-2019 学年，弗吉尼亚联邦大学医学院实施了一个移动友好型、学生发起的基于工作场所的评估（WBA）系统，为所有临床实习阶段的核心 EPAs 提供形成性反馈。学生需要在每个实习阶段要求进行特定数量的 Core EPA 评估。使用改良的 O-SCORE 量表（1 =“我必须做”到 4 =“我需要在房间里，以防万一”）来评估学习者的表现。应用概化理论评估评估的可概括性（或可靠性）。然后进行决策研究，以确定达到合理可靠性所需的评估数量。
共有 10680 次 WBA 对 220 名医学生进行了评估。大多数评分是在 EPA1（病史和体检）（n=3129；29%）和 EPA6（口头陈述）（n=2830；26%）上完成的。各 EPA 的平均得分相似（4 分制中的 3.5-3.6 分）。学生的方差从 3.5%到 8%不等，大部分方差归因于评分者（29.6%-50.3%）和其他未解释的因素。需要 25 到 63 次评估才能达到合理的可靠性（Phi >0.70）。
O-SCORE 在跨实习阶段使用时表现出适度的可靠性。这些发现突出了实施 Core EPAs 的 WBA 所面临的具体挑战，包括请求 WBA 的流程、评分者培训以及 O-SCORE 量表在医学生评估中的应用。
