Holst Daniel, Kowalewski Timothy M, White Lee W, Brand Timothy C, Harper Jonathan D, Sorensen Mathew D, Truong Mireille, Simpson Khara, Tanaka Alyssa, Smith Roger, Lendvay Thomas S
1 University of Washington School of Medicine , Seattle, Washington.
2 Department of Mechanical Engineering, University of Minnesota , Minneapolis, Minnesota.
J Endourol. 2015 Oct;29(10):1183-8. doi: 10.1089/end.2015.0104. Epub 2015 May 26.
Objective quantification of surgical skill is imperative as we enter a healthcare environment of quality improvement and performance-based reimbursement. The gold standard tools are infrequently used due to time-intensiveness, cost inefficiency, and lack of standard practices. We hypothesized that valid performance scores of surgical skill can be obtained through crowdsourcing.
Twelve surgeons of varying robotic surgical experience performed live porcine robot-assisted urinary bladder closures. Blinded video-recorded performances were scored by expert surgeon graders and by Amazon's Mechanical Turk crowdsourcing crowd workers using the Global Evaluative Assessment of Robotic Skills tool assessing five technical skills domains. Seven expert graders and 50 unique Mechanical Turkers (each paid $0.75/survey) evaluated each video. Global assessment scores were analyzed for correlation and agreement.
Six hundred Mechanical Turkers completed the surveys in less than 5 hours, while seven surgeon graders took 14 days. The duration of video clips ranged from 2 to 11 minutes. The correlation coefficient between the Turkers' and expert graders' scores was 0.95 and Cronbach's Alpha was 0.93. Inter-rater reliability among the surgeon graders was 0.89.
Crowdsourcing surgical skills assessment yielded rapid inexpensive agreement with global performance scores given by expert surgeon graders. The crowdsourcing method may provide surgical educators and medical institutions with a boundless number of procedural skills assessors to efficiently quantify technical skills for use in trainee advancement and hospital quality improvement.
随着我们进入一个以质量改进和绩效为基础的报销的医疗环境，对外科手术技能进行客观量化势在必行。由于耗时、成本低效以及缺乏标准做法，金标准工具很少被使用。我们假设可以通过众包获得有效的手术技能表现分数。
12名具有不同机器人手术经验的外科医生进行了实时猪机器人辅助膀胱闭合手术。由专家外科评分员和亚马逊的Mechanical Turk众包工作人员使用评估五个技术技能领域的机器人技能全球评估工具，对盲态视频记录的表现进行评分。7名专家评分员和50名不同的Mechanical Turk工作人员（每人每次调查支付0.75美元）评估每个视频。对全球评估分数进行相关性和一致性分析。
600名Mechanical Turk工作人员在不到5小时内完成了调查，而7名外科评分员则花费了14天。视频片段的时长从2分钟到11分钟不等。Turk工作人员和专家评分员分数之间的相关系数为0.95，克朗巴哈系数为0.93。外科评分员之间的评分者间信度为0.89。
众包手术技能评估与专家外科评分员给出的全球表现分数达成了快速且低成本的一致性。众包方法可能为外科教育工作者和医疗机构提供大量的程序技能评估人员，以有效地量化技术技能，用于学员晋升和医院质量改进。
