Ishman Stacey L, Benke James R, Johnson Kaalan Erik, Zur Karen B, Jacobs Ian N, Thorne Marc C, Brown David J, Lin Sandra Y, Bhatti Nasir, Deutsch Ellen S
Department of Otolaryngology–Head and Neck Surgery, Johns Hopkins School of Medicine, 601 N Caroline St, Sixth Floor, Baltimore, MD 21287, USA.
Arch Otolaryngol Head Neck Surg. 2012 Oct;138(10):916-22. doi: 10.1001/2013.jamaoto.115.
OBJECTIVES To confirm interrater reliability using blinded evaluation of a skills-assessment instrument to assess the surgical performance of resident and fellow trainees performing pediatric direct laryngoscopy and rigid bronchoscopy in simulated models. DESIGN Prospective, paired, blinded observational validation study. SUBJECTS Paired observers from multiple institutions simultaneously evaluated residents and fellows who were performing surgery in an animal laboratory or using high-fidelity manikins. The evaluators had no previous affiliation with the residents and fellows and did not know their year of training. INTERVENTIONS One- and 2-page versions of an objective structured assessment of technical skills (OSATS) assessment instrument composed of global and a task-specific surgical items were used to evaluate surgical performance. RESULTS Fifty-two evaluations were completed by 17 attending evaluators. The instrument agreement for the 2-page assessment was 71.4% when measured as a binary variable (ie, competent vs not competent) (κ = 0.38; P = .08). Evaluation as a continuous variable revealed a 42.9% percentage agreement (κ = 0.18; P = .14). The intraclass correlation was 0.53, considered substantial/good interrater reliability (69% reliable). For the 1-page instrument, agreement was 77.4% when measured as a binary variable (κ = 0.53, P = .0015). Agreement when evaluated as a continuous measure was 71.0% (κ = 0.54, P < .001). The intraclass correlation was 0.73, considered high interrater reliability (85% reliable). CONCLUSIONS The OSATS assessment instrument is an effective tool for evaluating surgical performance among trainees with acceptable interrater reliability in a simulator setting. Reliability was good for both the 1- and 2-page OSATS checklists, and both serve as excellent tools to provide immediate formative feedback on operational competency.
目的 通过对一项技能评估工具进行盲法评估，以确认评估者间的可靠性，该工具用于评估住院医师和专科培训医师在模拟模型中进行小儿直接喉镜检查和硬质支气管镜检查的手术操作表现。设计 前瞻性、配对、盲法观察性验证研究。对象 来自多个机构的配对观察者同时对在动物实验室或使用高保真人体模型进行手术的住院医师和专科培训医师进行评估。评估者此前与住院医师和专科培训医师无关联，且不知道他们的培训年份。干预措施 使用由整体和特定任务手术项目组成的1页和2页版本的客观结构化技术技能评估（OSATS）工具来评估手术操作表现。结果 17名主治评估者完成了52次评估。以二元变量（即胜任与否）衡量时，2页评估的工具一致性为71.4%（κ = 0.38；P = 0.08）。作为连续变量进行评估时，百分比一致性为42.9%（κ = 0.18；P = 0.14）。组内相关性为0.53，被认为具有较高的评估者间可靠性（69%可靠）。对于1页工具，以二元变量衡量时一致性为77.4%（κ = 0.53，P = 0.0015）。作为连续测量进行评估时一致性为71.0%（κ = 0.54，P < 0.001）。组内相关性为0.73，被认为具有高评估者间可靠性（85%可靠）。结论 OSATS评估工具是在模拟环境中评估学员手术操作表现的有效工具，具有可接受的评估者间可靠性。1页和2页的OSATS检查表的可靠性都很好，两者都是提供关于操作能力即时形成性反馈的优秀工具。
