Department of Surgery, Queen's University, Kingston, Ontario, Canada.
Department of Surgery, Queen's University, Kingston, Ontario, Canada.
J Surg Educ. 2019 May-Jun;76(3):872-880. doi: 10.1016/j.jsurg.2018.11.005. Epub 2018 Dec 17.
In competency-based medical education, progression between milestones requires reliable and valid methods of assessment. Surgery Tutor is an open-source motion tracking platform developed to objectively assess technical proficiency during open soft-tissue tumor resections in a simulated setting. The objective of our study was to provide evidence in support of construct validity of the scores obtained by Surgery Tutor. We hypothesized that Surgery Tutor would discriminate between novice, intermediate, and experienced operators.
Thirty participants were assigned to novice, intermediate, or experienced groups, based on the number of prior soft-tissue resections performed. Each participant resected 2 palpable and 2 nonpalpable lesions from a soft-tissue phantom. Surgery Tutor was used to track hand and instrument motions, number of tumor breaches, and time to perform each resection. Mass of excised specimens and margin status were also recorded.
Surgery Tutor scores demonstrated "moderate" to "good" internal structure (test-retest reliability) for novice, intermediate, and experienced groups (interclass correlation coefficient = 0.596, 0.569, 0.737; p < 0.001). Evidence in support of construct validity (consequences) was demonstrated by comparing scores of novice, intermediate, and experienced participantsfor number of hand and instrument motions (690 ± 190, 597 ± 169, 469 ± 110; p < 0.001), number of tumor breaches (29 ± 34, 16 ± 11, 9 ± 6; p < 0.001), time per resection (677 ± 331 seconds, 561 ± 210 seconds, 449 ± 148 seconds; p < 0.001), mass of completely excised specimens (22 ± 7g, 21 ± 11g, 17 ± 6 g; p = 0.035), and rate of positive margin (68%, 50%, 28%; p < 0.001). There was "strong" and "moderate" relationships between motion scores and Objective Structured Assessment of Technical Skill scores, and time per resection and Objective Structured Assessment of Technical Skill scores respectively (r = -0.60, p < 0.001; r = -0.54, p < 0.001).
Surgery Tutor scores demonstrate evidenceof construct validity with regards to good internal structure, consequences, and relationship to other variables in the assessment of technical proficiency duringopen soft-tissue tumor resections in a simulated setting. Utilization of Surgery Tutor can provide formative feedback and objective assessment of surgical proficiency in a simulated setting.
在基于能力的医学教育中，要达到里程碑式的进展，需要可靠和有效的评估方法。Surgery Tutor 是一个开源的运动跟踪平台，旨在客观评估模拟环境下开放性软组织肿瘤切除术中的技术熟练程度。我们的研究目的是提供支持 Surgery Tutor 获得的分数的结构有效性的证据。我们假设 Surgery Tutor 将区分新手、中级和经验丰富的操作人员。
根据进行的软组织切除术数量，将 30 名参与者分为新手、中级或经验丰富组。每位参与者从软组织模型中切除 2 个可触及和 2 个不可触及的病变。使用 Surgery Tutor 跟踪手和器械的运动、肿瘤破裂的次数以及每次切除的时间。切除标本的质量和边缘状态也被记录下来。
Surgery Tutor 分数对于新手、中级和经验丰富组的内部结构（重测信度）显示为“中等”至“良好”（组内相关系数=0.596、0.569、0.737；p<0.001）。通过比较新手、中级和经验丰富参与者的分数，证明了结构有效性（后果）的证据，包括手和器械运动次数（690±190、597±169、469±110；p<0.001）、肿瘤破裂次数（29±34、16±11、9±6；p<0.001）、每次切除的时间（677±331 秒、561±210 秒、449±148 秒；p<0.001）、完全切除标本的质量（22±7g、21±11g、17±6g；p=0.035）和阳性边缘率（68%、50%、28%；p<0.001）。运动评分与客观结构化技术技能评分之间存在“强”和“中等”的关系，而每次切除的时间与客观结构化技术技能评分之间存在“中等”关系（r=-0.60，p<0.001；r=-0.54，p<0.001）。
Surgery Tutor 分数在模拟环境下开放性软组织肿瘤切除术中的技术熟练程度评估方面，表现出结构有效性的证据，包括良好的内部结构、后果以及与其他变量的关系。在模拟环境中使用 Surgery Tutor 可以提供形成性反馈和对手术熟练程度的客观评估。
