Graduate Medical Education, Virginia Mason Medical Center, Mailstop H8-GME, 1100 9th Ave., Seattle, WA, 98101, USA.
Department of Surgery, Indiana University, Bloomington, IN, USA.
Surg Endosc. 2017 Dec;31(12):5094-5100. doi: 10.1007/s00464-017-5574-1. Epub 2017 Apr 25.
Faculty experts (FE) and crowd workers (CW) can assess technical skill, but assessment of operative technique has not been explored. We sought to evaluate if CW could be taught to assess completion of the critical view of safety (CVS) in laparoscopic cholecystectomy.
We prepared 160 blinded, surgical videos of laparoscopic cholecystectomy from public domain websites. Videos were edited to ≤60 s, ending when a structure was cut/clipped. CW analyzed videos using Global Objective Assessment of Laparoscopic Skills (GOALS) and CVS criteria assessment tools after watching an instructional tutorial. Ten videos were randomly selected from each performance quartile based on GOALS. Five FE rated the 40 videos using GOALS and CVS. Linear mixed effects models derived average CW and FE ratings for GOALS and CVS for each video. Spearman correlation coefficients (SCC) were used to assess the degree of correlation between performance measures. Satisfactory completion of the CVS was defined as scoring an average CVS ≥ 5. Videos with an average GOALS ≥ 15 were considered top technical performers.
A high degree of correlation was seen between all performance measures: CVS ratings between CW and FE, SCC 0.89 (p < 0.001); GOALS and CVS ratings SCC 0.77 (p < 0.001) for CW, and SCC 0.71 (p < 0.001) for FE. Sixteen videos were assigned top technical performer ratings by both CW and FE but the average CVS was inadequate (3.8 and 3.6, respectively), and the percentage of satisfactory CVS ≥ 5 was 12.5%.
A high degree of correlation was found between CW and FE in assessment of the CVS. However, in this video analysis, high technical performers did not achieve a complete CVS in most cases. Educating CW to assess operative technique for the identification of low or average performers is feasible and may broaden the application of this assessment and feedback tool.
教师专家（FE）和众包工人（CW）可以评估技术技能，但尚未探索手术技术的评估。我们试图评估 CW 是否可以接受培训以评估腹腔镜胆囊切除术的关键安全视图（CVS）的完成情况。
我们从公共领域网站准备了 160 个腹腔镜胆囊切除术的盲法、手术视频。视频剪辑至≤60s，当切割/剪断某个结构时结束。CW 在观看教学教程后，使用全球腹腔镜技能评估（GOALS）和 CVS 标准评估工具分析视频。根据 GOALS ，从每个表现四分位数中随机选择 10 个视频。5 位 FE 使用 GOALS 和 CVS 对 40 个视频进行评分。线性混合效应模型得出了每个视频的 CW 和 FE 对 GOALS 和 CVS 的平均评分。使用 Spearman 相关系数（SCC）评估绩效评估之间的相关性程度。令人满意的 CVS 完成定义为平均 CVS 评分≥5。平均 GOALS 评分≥15 的视频被认为是顶级技术表演者。
所有表现测量之间都存在高度相关性：CW 和 FE 之间的 CVS 评分，SCC 0.89（p<0.001）；CW 的 GOALS 和 CVS 评分 SCC 0.77（p<0.001）和 FE 的 SCC 0.71（p<0.001）。CW 和 FE 都将 16 个视频分配为顶级技术表演者评分，但平均 CVS 不足（分别为 3.8 和 3.6），满意的 CVS≥5 的百分比为 12.5%。
CW 和 FE 在 CVS 评估方面发现了高度的相关性。然而，在这项视频分析中，在大多数情况下，高技能表演者并没有完全实现 CVS。教育 CW 评估手术技术以识别低或平均表现者是可行的，并且可能扩大了这种评估和反馈工具的应用。
