Polin Michael R, Siddiqui Nazema Y, Comstock Bryan A, Hesham Helai, Brown Casey, Lendvay Thomas S, Martino Martin A
Division of Urogynecology, Department of Obstetrics and Gynecology, Duke University, Durham, NC.
Division of Urogynecology, Department of Obstetrics and Gynecology, Duke University, Durham, NC.
Am J Obstet Gynecol. 2016 Nov;215(5):644.e1-644.e7. doi: 10.1016/j.ajog.2016.06.033. Epub 2016 Jun 27.
Robotic-assisted gynecologic surgery is common, but requires unique training. A validated assessment tool for evaluating trainees' robotic surgery skills is Robotic-Objective Structured Assessments of Technical Skills.
We sought to assess whether crowdsourcing can be used as an alternative to expert surgical evaluators in scoring Robotic-Objective Structured Assessments of Technical Skills.
The Robotic Training Network produced the Robotic-Objective Structured Assessments of Technical Skills, which evaluate trainees across 5 dry lab robotic surgical drills. Robotic-Objective Structured Assessments of Technical Skills were previously validated in a study of 105 participants, where dry lab surgical drills were recorded, de-identified, and scored by 3 expert surgeons using the Robotic-Objective Structured Assessments of Technical Skills checklist. Our methods-comparison study uses these previously obtained recordings and expert surgeon scores. Mean scores per participant from each drill were separated into quartiles. Crowdworkers were trained and calibrated on Robotic-Objective Structured Assessments of Technical Skills scoring using a representative recording of a skilled and novice surgeon. Following this, 3 recordings from each scoring quartile for each drill were randomly selected. Crowdworkers evaluated the randomly selected recordings using Robotic-Objective Structured Assessments of Technical Skills. Linear mixed effects models were used to derive mean crowdsourced ratings for each drill. Pearson correlation coefficients were calculated to assess the correlation between crowdsourced and expert surgeons' ratings.
In all, 448 crowdworkers reviewed videos from 60 dry lab drills, and completed a total of 2517 Robotic-Objective Structured Assessments of Technical Skills assessments within 16 hours. Crowdsourced Robotic-Objective Structured Assessments of Technical Skills ratings were highly correlated with expert surgeon ratings across each of the 5 dry lab drills (r ranging from 0.75-0.91).
Crowdsourced assessments of recorded dry lab surgical drills using a validated assessment tool are a rapid and suitable alternative to expert surgeon evaluation.
机器人辅助妇科手术很常见，但需要独特的培训。用于评估学员机器人手术技能的一种经过验证的评估工具是机器人技术技能客观结构化评估。
我们试图评估在对机器人技术技能客观结构化评估进行评分时，众包是否可以用作专家手术评估者的替代方法。
机器人培训网络制作了机器人技术技能客观结构化评估，该评估通过5项干式实验室机器人手术操作对学员进行评估。机器人技术技能客观结构化评估先前在一项针对105名参与者的研究中得到验证，在该研究中，干式实验室手术操作被录制、去除身份标识，并由3名专家外科医生使用机器人技术技能客观结构化评估清单进行评分。我们的方法比较研究使用这些先前获得的记录和专家外科医生的评分。每个操作中每位参与者的平均得分被分为四分位数。使用熟练和新手外科医生的代表性记录，对众包工作者进行机器人技术技能客观结构化评估评分的培训和校准。在此之后，从每个操作的每个评分四分位数中随机选择3个记录。众包工作者使用机器人技术技能客观结构化评估对随机选择的记录进行评估。使用线性混合效应模型得出每个操作的众包平均评分。计算皮尔逊相关系数以评估众包评分与专家外科医生评分之间的相关性。
共有448名众包工作者查看了来自60项干式实验室操作的视频，并在16小时内完成了总共2517次机器人技术技能客观结构化评估。在5项干式实验室操作中的每一项中，众包的机器人技术技能客观结构化评估评分与专家外科医生评分高度相关（r值范围为0.75 - 0.91）。
使用经过验证的评估工具对录制的干式实验室手术操作进行众包评估是专家外科医生评估的一种快速且合适的替代方法。
