Dwyer Tim, Slade Shantz Jesse, Kulasegaram Kulamakan Mahan, Chahal Jaskarndip, Wasserstein David, Schachar Rachel, Devitt Brian, Theodoropoulos John, Hodges Brian, Ogilvie-Harris Darrell
Women's College Hospital, Toronto, Ontario, Canada; Mt. Sinai Hospital, Toronto, Ontario, Canada.
University of Toronto, Toronto, Ontario, Canada.
Arthroscopy. 2016 Dec;32(12):2572-2581.e3. doi: 10.1016/j.arthro.2016.05.037. Epub 2016 Jul 27.
The purpose of this study was to determine if the use of an Objective Structured Assessment of Technical skill (OSATS), using dry models, would be a valid method of assessing residents' ability to perform sports medicine procedures after training in a competency-based model.
Over 18 months, 27 residents (19 junior [postgraduate year (PGY) 1-3] and 8 senior [PGY 4-5]) sat the OSATS after their rotation, in addition to 14 sports medicine staff and fellows. Each resident was provided a list of 10 procedures in which they were expected to show competence. At the end of the rotation, each resident undertook an OSATS composed of 6 stations sampled from the 10 procedures using dry models-faculty used the Arthroscopic Surgical Skill Evaluation Tool (ASSET), task-specific checklists, as well as an overall 5-point global rating scale (GRS) to score each resident. Each procedure was videotaped for blinded review.
The overall reliability of the OSATS (0.9) and the inter-rater reliability (0.9) were both high. A significant difference by year in training was seen for the overall GRS, the total ASSET score, and the total checklist score, as well as for each technical procedure (P < .001). Further analysis revealed a significant difference in the total ASSET score between junior (mean 18.4, 95% confidence interval [CI] 16.8 to 19.9) and senior residents (24.2, 95% CI 22.7 to 25.6), senior residents and fellows (30.1, 95% CI 28.2 to 31.9), as well as between fellows and faculty (37, 95% CI 36.1 to 27.8) (P < .05).
The results of this study show that an OSATS using dry models shows evidence of validity when used to assess performance of technical procedures after a sports medicine rotation. However, junior residents were not able to perform as well as senior residents, suggesting that overall surgical experience is as important as intensive teaching.
As postgraduate medical training shifts to a competency-based model, methods of assessing performance of technical procedures become necessary.
本研究旨在确定使用客观结构化技术技能评估（OSATS）并采用干式模型，是否是评估住院医师在基于胜任力模型培训后执行运动医学手术能力的有效方法。
在18个月的时间里，27名住院医师（19名初级住院医师[研究生第1 - 3年]和8名高级住院医师[研究生第4 - 5年]）在轮转结束后参加了OSATS评估，另外还有14名运动医学工作人员和研究员。为每位住院医师提供了一份包含10项手术的清单，期望他们在这些手术中展现出胜任能力。在轮转结束时，每位住院医师进行了一次OSATS评估，该评估由从10项手术中抽取的6个站组成，使用干式模型——教员使用关节镜手术技能评估工具（ASSET）、特定任务清单以及一个整体的5分制全球评分量表（GRS）对每位住院医师进行评分。每项手术都进行了录像以便进行盲法评审。
OSATS的整体信度（0.9）和评分者间信度（0.9）都很高。在整体GRS、ASSET总分、清单总分以及每项技术手术方面，不同培训年份存在显著差异（P < .001）。进一步分析显示，初级住院医师（平均18.4，95%置信区间[CI] 16.8至19.9）与高级住院医师（24.2，95% CI 22.7至25.6）、高级住院医师与研究员（30.1，95% CI 28.2至31.9）以及研究员与教员（37，95% CI 36.1至27.8）之间的ASSET总分存在显著差异（P < .05）。
本研究结果表明，使用干式模型的OSATS在用于评估运动医学轮转后技术手术的操作表现时具有效度证据。然而，初级住院医师的表现不如高级住院医师，这表明整体手术经验与强化教学同样重要。
随着研究生医学培训转向基于胜任力的模型，评估技术手术操作表现的方法变得必要。
