Wellspan Health, York, Pennsylvania.
Division of Education, American College of Surgeons, Chicago Illinois.
J Surg Educ. 2022 Nov-Dec;79(6):e194-e201. doi: 10.1016/j.jsurg.2022.07.007. Epub 2022 Jul 25.
The objective assessment of technical skills of junior residents is essential in implementing competency-based training and providing specific feedback regarding areas for improvement. An innovative assessment that can be easily implemented by training programs nationwide has been developed by expert surgeon educators under the aegis of the American College of Surgeons (ACS) Division of Education. This assessment, ACS Objective Assessment of Skills in Surgery (ACS OASIS) uses eight stations to address technical skills important for junior residents within the domains of laparoscopic appendectomy, excision of lipoma, central line placement, laparoscopic cholecystectomy, trocar placement, exploratory laparotomy, repair of enterotomy, and tube thoracostomy. The purpose of this study was to implement ACS OASIS at a number of sites to study its psychometric rigor.
The ACS OASIS was pre-piloted at two programs to establish feasibility and to gather information regarding implementation. Each skills station was 12 minutes long, and the faculty completed a checklist with 5 to 15 items, and a global assessment scale. The study was then repeated at three pilot sites and included 29 junior residents who were assessed by a total of 44 faculty. Psychometric data for the stations and checklists were collected and analyzed.
The pre-pilot sites were Geisinger and University of Tennessee Knoxville.Data were gathered from pilot sites that included Wellspan Health, Duke University, and University of California Los Angeles.
The mean checklist score for all learners was 76% (IQR of 66%-85%). The average global rating was 3.36 on a 5-point scale with a standard deviation of 0.56. The overall cut score derived using the borderline group method was at 68% with 34% of performances requiring remediation. Using this criterion, the average number of stations that were completed by each learner without need for remediation was five.The station discrimination index ranged from 0.27 to 0.65 (all above the threshold of 0.25), demonstrating solid psychometric characteristics at the station level. The internal-consistency reliability was 0.76 with SEM of 5.8%. The inter-rater reliability (intraclass correlation) was high at 0.73 with general agreement of 79% between the two raters. The station discrimination was at 0.45 (range of 0.27 to 0.65) indicating a high level of differentiation between high and low performers. Using the generalizability theory, the G-coefficient reliability was at 0.72 with the reliability projection flattening after 8 stations. Overall, 75% to 82% the faculty and learners rated ACS OASIS as realistic and beneficial.
ACS OASIS is a psychometrically sound technical skills assessment tool that can provide useful information for feedback to junior residents and support efforts to remediate gaps in performance.
对初级住院医师的技术技能进行客观评估对于实施基于能力的培训和提供有关改进领域的具体反馈至关重要。在外科医师教育专家的倡导下，美国外科医师学院 (ACS) 教育司开发了一种创新的评估方法，该方法可以由全国的培训计划轻松实施。这种评估方法，即 ACS 手术技能客观评估 (ACS OASIS) 使用八个站点来解决初级住院医师在腹腔镜阑尾切除术、脂肪瘤切除术、中央线放置、腹腔镜胆囊切除术、套管放置、剖腹探查术、肠切开术修复和胸腔引流管放置等领域的重要技术技能。本研究的目的是在多个站点实施 ACS OASIS，以研究其心理测量严谨性。
ACS OASIS 在两个项目中进行了预试点，以确定可行性并收集有关实施的信息。每个技能站的持续时间为 12 分钟，教员完成了一份包含 5 到 15 个项目的检查表，以及一份全球评估量表。然后，该研究在三个试点站点重复进行，包括 29 名初级住院医师，由 44 名教员进行评估。收集并分析了各站和检查表的心理测量数据。
预试点地点为 Geisinger 和田纳西大学诺克斯维尔分校。数据来自包括 Wellspan Health、杜克大学和加利福尼亚大学洛杉矶分校在内的试点站点。
所有学习者的平均检查表得分为 76%（66%-85%的 IQR）。平均全球评分在 5 分制上为 3.36，标准差为 0.56。使用边界组方法得出的总切割分数为 68%，其中 34%的表现需要补救。使用此标准，每个学习者无需补救即可完成的平均站数为 5 个。站区分指数范围为 0.27 至 0.65（均高于 0.25 的阈值），表明站级具有可靠的心理测量特征。内部一致性信度为 0.76，标准误为 5.8%。评分者间可靠性（组内相关系数）很高，为 0.73，两位评分者之间的总体一致性为 79%。站区分度为 0.45（范围为 0.27 至 0.65），表明高绩效者和低绩效者之间有很高的区分度。使用概化理论，G 系数信度为 0.72，8 站后可靠性预测趋于平稳。总体而言，75%至 82%的教员和学习者认为 ACS OASIS 具有现实性和有益性。
ACS OASIS 是一种可靠的心理测量技术技能评估工具，可为初级住院医师提供反馈的有用信息，并支持努力纠正绩效差距。
