Friedman Zeev, Katznelson Rita, Devito Isabel, Siddiqui Mughina, Chan Vincent
Department of Anesthesia, Mount Sinai Hospital, Toronto, Ontario, Canada.
Reg Anesth Pain Med. 2006 Jul-Aug;31(4):304-10. doi: 10.1016/j.rapm.2006.04.006.
Demand is growing for objective assessment of manual skills and competencies of invasive procedures. The aim of this study was to validate an objective tool for assessing residents' skill in performing epidural anesthesia by use of a global assessment scale and a 3-scale, 27-stage checklist. We wish to demonstrate that this tool can differentiate operators with different levels of training.
Second-year anesthesia residents were recruited. Their previous experience was assessed by questionnaire. They were repeatedly videotaped performing epidural anesthesia over a 6-month period. Videotaping was done in a blinded manner that masked the identity and level of training of the residents. Three blinded, independent examiners evaluated each session by use of a specifically devised assessment tool that consisted of a global rating scale and a 3-scale, 27-stage checklist to judge the skill level and grade the videotaped sessions.
Twenty-one sessions by 6 residents were videotaped over 6 months. Interrater reliability for the different checklist and global-rating form items shows moderate to high degree of agreement for most stages. Total scores demonstrate almost perfect agreement (kappa/ICC +/- SE = 0.90 +/- 0.03 and 0.83 +/- 0.13, respectively; P < .0001) between examiners. To test whether higher total scores are associated with greater experience, a series of repeated-measures ANCOVAs were performed. In both the global-rating form and the checklist, a significant relation between total scores and epidurals done was found to exist (checklist: P < .0001; global rating: P < .0001).
The results of our study show that scores on a system that consists of a global-rating form and a task-specific checklist had a significant relation to the number of epidural insertions performed (i.e., experience). The interrater reliability of these assessment tools was very strong. Evaluation of technical skills by an objective tool under direct observation, as opposed to laboratory setting, may create a more reliable standard of assessment. Furthermore, residency programs could use these evaluations to identify deficiencies in teaching programs and trainees who require extra instruction.
对侵入性操作的手动技能和能力进行客观评估的需求日益增长。本研究的目的是通过使用整体评估量表和一个三等级、27阶段的检查表来验证一种用于评估住院医师进行硬膜外麻醉技能的客观工具。我们希望证明该工具能够区分不同培训水平的操作者。
招募二年级麻醉住院医师。通过问卷调查评估他们之前的经验。在6个月的时间里，对他们进行硬膜外麻醉的操作过程进行多次录像。录像以盲法进行，掩盖了住院医师的身份和培训水平。三名盲法、独立的检查者使用一种专门设计的评估工具对每次操作进行评估，该工具包括一个整体评分量表和一个三等级、27阶段的检查表，以判断技能水平并对录像操作进行评分。
在6个月内对6名住院医师的21次操作进行了录像。不同检查表项目和整体评分表项目的评分者间信度在大多数阶段显示出中度到高度的一致性。检查者之间的总分显示出几乎完美的一致性（kappa/ICC±SE分别为0.90±0.03和0.83±0.13；P<.0001）。为了测试更高的总分是否与更多经验相关，进行了一系列重复测量的协方差分析。在整体评分表和检查表中，均发现总分与完成的硬膜外麻醉次数之间存在显著关系（检查表：P<.0001；整体评分：P<.0001）。
我们的研究结果表明，由整体评分表和特定任务检查表组成的系统的得分与进行硬膜外穿刺的次数（即经验）有显著关系。这些评估工具的评分者间信度非常高。与实验室环境不同，在直接观察下通过客观工具对技术技能进行评估可能会创建一个更可靠的评估标准。此外，住院医师培训项目可以利用这些评估来识别教学项目中的不足以及需要额外指导的学员。
