Department of Surgery, Western University, London, Ontario, Canada.
Faculty of Education, Queen's University, Kingston, Ontario, Canada.
J Surg Educ. 2021 Nov-Dec;78(6):2052-2062. doi: 10.1016/j.jsurg.2021.05.004. Epub 2021 Jun 3.
Demonstrated competence through frequent assessment is an expected goal for progressive development in competency-based medical education curricula. The Objective Structured Assessment of Technical Skill (OSATS) is considered a valid method of formative assessment, but in few instances have standards been set for determining competence. The present study used borderline regression methods to examine standard setting of performance on a complex technical task with novices assessed using an OSATS checklist.
This was a single institution prospective single arm experimental design study. Participants were 58 non-medical undergraduate students with no previous surgical experience, who observed a computer-based training module on end-to-side vascular anastomosis. Subsequently, participants were provided two 20-minute training sessions, two weeks apart where they received expert feedback whilst performing the task on a low-fidelity model. After each training session, participants completed the task unaided. Sessions were recorded and assessed using an OSATS checklist retrospectively by experts.
Paired t-test analyses indicate that for both the checklist total score (t(52) = 8.05, p < 0.001) and the global rating score (t(53) = 8.15, p < 0.001), individuals performed significantly better in Phase 2. Borderline regression analyses indicated that in Phase 1 (R = .60) and Phase 2 (R = .75), the OSATS checklist could adequately capture variation in performance in novices. Further, the checklist could reliably classify novices at three of the five global rating performance levels. Pass rates determined by regression equations improved from Phase 1 to Phase 2 on all global rating levels.
With the increasing focus on competency-based medical education, it is imperative that training programs have the capacity to accurately assess outcomes and set minimum performance standards. Borderline regression methods can accurately differentiate novice learners of varying performance levels before and after training on a complex technical skill task using an OSATS checklist.
在以能力为基础的医学教育课程中，通过频繁评估来展示能力是一个预期的目标。客观结构化临床技能评估（OSATS）被认为是一种有效的形成性评估方法，但在确定能力方面很少设定标准。本研究使用边界回归方法，检查使用 OSATS 检查表评估新手复杂技术任务表现的标准设定。
这是一项单机构前瞻性单臂实验设计研究。参与者是 58 名没有手术经验的非医学本科学生，他们观察了基于计算机的端侧血管吻合术培训模块。随后，参与者在两周内接受了两次 20 分钟的培训，在此期间，他们在低保真模型上执行任务时获得了专家反馈。每次培训后，参与者都在没有帮助的情况下完成任务。使用 OSATS 检查表记录和评估课程，由专家进行回顾性评估。
配对 t 检验分析表明，对于检查表总分（t(52) = 8.05，p < 0.001）和全球评分（t(53) = 8.15，p < 0.001），个体在第 2 阶段的表现明显更好。边界回归分析表明，在第 1 阶段（R =.60）和第 2 阶段（R =.75），OSATS 检查表可以充分捕捉新手表现的变化。此外，检查表可以可靠地将新手分为五个全球评分表现水平中的三个水平。在所有全球评分水平上，通过回归方程确定的通过率从第 1 阶段提高到第 2 阶段。
随着以能力为基础的医学教育的关注度不断提高，培训计划必须有能力准确评估结果并设定最低绩效标准。边界回归方法可以在使用 OSATS 检查表对复杂技术任务进行培训前后，准确区分不同表现水平的新手学习者。
