Department of Otolaryngology-Head and Neck Surgery, Johns Hopkins School of Medicine, Baltimore, Maryland 21287, USA.
Laryngoscope. 2010 Nov;120(11):2294-300. doi: 10.1002/lary.21067.
To develop and pilot test an objective assessment of technical skills instrument for evaluation of residents' surgical performance of pediatric direct laryngoscopy and rigid bronchoscopy, with emphasis on feasibility, validity, and interrater agreement.
Prospective, unblinded educational quality-improvement project combining paired and unpaired observations.
Objective Structured Assessment of Technical Skills (OSATS) evaluation instruments were developed from detailed faculty input. Two instruments were created: a surgical checklist and a global assessment of surgical performance. The instruments were tested in a simulation bronchoscopy course and in the operative suite. Paired observations were used to determine interrater agreement, whereas multiple evaluations were used to calculate construct validity and internal consistency.
Forty-four assessments were completed for 19 residents. Seven faculty members evaluated residents as they performed laryngoscopy and bronchoscopy in an animal simulation laboratory and in the operating room. The evaluation tool was found to be feasible. The interrater agreement for both instruments was significant at 80.4% to 84.6% (P ≤.0001). Construct validity was confirmed with increasing mean global and task specific scores by postgraduate year (P <.0001). Internal consistency, measured with Cronbach alpha, was high at 0.968.
This pilot study suggests that a reliable and valid instrument for objective evaluation of surgical competency can be developed for pediatric direct laryngoscopy and rigid bronchoscopy. This instrument can be used for formative and summative feedback of operative performance. In addition, it was easy to use and valid in a limited evaluation; however, larger studies are required to validate its utility.
开发并初步测试一种用于评估住院医师行小儿直接喉镜和硬性支气管镜手术技能的客观评估工具，重点关注其可行性、有效性和评分者间一致性。
前瞻性、非盲的教育质量改进项目，结合配对和非配对观察。
客观结构化临床技能评估（OSATS）评估工具是根据详细的教师输入开发的。创建了两个工具：手术检查表和手术绩效综合评估。该工具在模拟支气管镜课程和手术室中进行了测试。配对观察用于确定评分者间一致性，而多次评估用于计算结构有效性和内部一致性。
对 19 名住院医师进行了 44 次评估。7 名教师在动物模拟实验室和手术室观察住院医师进行喉镜和支气管镜检查。评估工具被发现具有可行性。两种工具的评分者间一致性均达到 80.4%至 84.6%（P≤.0001）。随着研究生年级的增加，整体和特定任务的平均得分呈递增趋势，从而验证了结构有效性（P<.0001）。内部一致性，用克朗巴赫 α 衡量，高达 0.968。
这项初步研究表明，一种用于小儿直接喉镜和硬性支气管镜手术技能客观评估的可靠且有效的工具可以开发出来。该工具可用于手术表现的形成性和总结性反馈。此外，它易于使用且在有限的评估中有效；然而，需要更大的研究来验证其效用。
