Cheon Stephanie, de Jager Cornelis, Egan Rylan, Bona Mark, Law Christine
School of Medicine, Faculty of Health Sciences, Queen's University, Kingston, Ont.; Department of Ophthalmology, Queen's University and Kingston Health Sciences Centre, Kingston, Ont.
Department of Ophthalmology, Queen's University and Kingston Health Sciences Centre, Kingston, Ont.
Can J Ophthalmol. 2020 Oct;55(5):382-390. doi: 10.1016/j.jcjo.2020.05.005. Epub 2020 Jun 24.
To evaluate the accuracy of ophthalmology residents' self-assessment and peer assessment of surgical skills in a simulation setting.
Simulation laboratory assessment.
Ophthalmology residents novice to cataract surgery.
A modified International Council of Ophthalmology's Ophthalmology Surgical Competency Assessment Rubric: Phacoemulsification structured assessment tool for simulated cataract surgery was established by conventional Delphi method. Residents completed 10 independent simulated surgeries that were video-recorded. Two experts graded the videos using the assessment tool. Participants performed self-assessment of their own 10 videos, and peer assessment of 10 of their peers' videos.
Nine cataract surgery experts provided feedback and modifications for the assessment tool. Agreement for the first round of the Delphi method ranged from 55.56% to 100%. Second round agreement was 80% or greater for all answers. The final assessment tool comprised (i) 4 procedural items scored from 0 (not performed) to 7 (competent), and (ii) a global rating scale (GRS) requiring yes/no answers to 4 performance-related questions. Eight residents participated in the study. There was excellent expert inter-rater reliability intraclass correlation ((ICC) = 0.844, 0.875, 0.809, 0.844) and fair to excellent inter-rater reliability between expert and peer scores (ICC = 0.702, 0.831, 0.521, 0.423), but systematic disagreement (ICC = -0.428, -0.038) or poor inter-rater reliability (ICC = 0.298, 0.362) between expert and self-scores. There was poor agreement for all GRS questions (κ statistic < 0.40) except 2 comparisons.
In the simulation setting, experts were able to reliably assess trainees' performance using the assessment tool. Participants demonstrated inconsistency in assessing their own skills; however, they were adequate at assessing their peers' overall performance.
评估眼科住院医师在模拟环境中对手术技能的自我评估和同伴评估的准确性。
模拟实验室评估。
初次接触白内障手术的眼科住院医师。
采用传统的德尔菲法建立了改良的国际眼科理事会眼科手术能力评估量表：用于模拟白内障手术的超声乳化结构化评估工具。住院医师完成10次独立的模拟手术，并进行视频录制。两名专家使用该评估工具对视频进行评分。参与者对自己的10个视频进行自我评估，并对10名同伴的视频进行同伴评估。
9名白内障手术专家对评估工具提供了反馈并进行了修改。第一轮德尔菲法的一致性范围为55.56%至100%。第二轮所有答案的一致性均达到80%或更高。最终评估工具包括：（i）4个程序项目，评分从0（未执行）到7（胜任）；（ii）一个整体评分量表（GRS），需要对4个与表现相关的问题回答“是”或“否”。8名住院医师参与了研究。专家之间的评分者信度极佳（组内相关系数（ICC）=0.844、0.875、0.809、0.844），专家与同伴评分之间的评分者信度良好至极佳（ICC=0.702、0.831、0.521、0.423），但专家与自我评分之间存在系统性差异（ICC=-0.428、-0.038）或评分者信度较差（ICC=0.298、0.362）。除2项比较外，所有GRS问题的一致性均较差（κ统计量<0.40）。
在模拟环境中，专家能够使用评估工具可靠地评估学员的表现。参与者在评估自己的技能时表现出不一致性；然而，他们在评估同伴的整体表现方面表现尚可。
