Copenhagen Academy for Medical Education and Simulation (CAMES), The Capital Region of Denmark, Copenhagen, Denmark.
Copenhagen Academy for Medical Education and Simulation (CAMES), The Capital Region of Denmark, Copenhagen, Denmark; Department of Vascular Surgery, Rigshospitalet, Copenhagen, Denmark.
Eur J Vasc Endovasc Surg. 2020 May;59(5):767-774. doi: 10.1016/j.ejvs.2020.01.021. Epub 2020 Feb 20.
The aims of this study were to develop a procedure specific assessment tool for open abdominal aortic aneurysm (AAA) repair, gather validity evidence for the tool and establish a pass/fail standard.
Validity was studied based on the contemporary framework by Messick. Three vascular surgeons experienced in open AAA repair and an expert in assessment and validation within medical education developed the OPEn aortic aneurysm Repair Assessment of Technical Expertise (OPERATE) tool. Vascular surgeons with varying experiences performed open AAA repair in a standardised simulation based setting. All procedures were video recorded with the faces anonymised and scored independently by three experts in a mutual blinded setup. The Angoff standard setting method was used to establish a credible pass/fail score.
Sixteen novices and nine experienced open vascular surgeons were enrolled. The OPERATE tool achieved high internal consistency (Cronbach's alpha .92) and inter-rater reliability (Cronbach's alpha .95) and was able to differentiate novices and experienced surgeons with mean scores (higher score is better) of 13.4 ± 12 and 25.6 ± 6, respectively (p = .01). The pass/fail score was set high (27.7). One novice passed the test while six experienced surgeons failed.
Validity evidence was established for the newly developed OPERATE tool and was able to differentiate between novices and experienced surgeons providing a good argument that this tool can be used for both formative and summative assessment in a simulation based environment. The high pass/fail score emphasises the need for novices to train in a simulation based environment up to a certain level of competency before apprenticeship training in the clinical environment under the tutelage of a supervisor. Familiarisation with the simulation equipment must be ensured before performance is assessed as reflected by the low scores in the experienced group's first attempt.
本研究旨在开发一种针对开放式腹主动脉瘤（AAA）修复的特定程序评估工具，为该工具收集有效性证据并建立通过/失败标准。
基于当代 Messick 框架进行有效性研究。三位在开放式 AAA 修复方面经验丰富的血管外科医生和一位在医学教育评估和验证方面的专家共同开发了 OPEn aortic aneurysm Repair Assessment of Technical Expertise（OPERATE）工具。具有不同经验的血管外科医生在标准化的模拟环境中进行开放式 AAA 修复。所有手术均进行视频记录，脸部匿名化，并由三位专家在相互盲法设置下进行独立评分。采用 Angoff 标准设定法建立可信的通过/失败分数。
纳入了 16 名新手和 9 名经验丰富的开放式血管外科医生。OPERATE 工具具有较高的内部一致性（Cronbach's alpha.92）和评分者间可靠性（Cronbach's alpha.95），能够区分新手和经验丰富的外科医生，其平均得分（得分越高越好）分别为 13.4±12 和 25.6±6（p=.01）。通过/失败分数设定较高（27.7）。一名新手通过了测试，而六名经验丰富的外科医生失败了。
为新开发的 OPERATE 工具建立了有效性证据，并能够区分新手和经验丰富的外科医生，这有力地证明了该工具可用于模拟环境中的形成性和总结性评估。高通过/失败分数强调了新手在临床环境中在导师的指导下进行学徒培训之前，需要在模拟环境中进行一定水平的培训。在进行绩效评估之前，必须确保熟悉模拟设备，因为这反映了经验丰富组在首次尝试中的得分较低。
