Department of Surgery, University of Texas Southwestern Medical Center, 5323 Harry Hines Blvd, Dallas, TX, 75390-9159, USA.
University of Texas Southwestern Simulation Center, 2001 Inwood Road, Dallas, TX, 75390-9092, USA.
Surg Endosc. 2023 Jan;37(1):402-411. doi: 10.1007/s00464-022-09509-y. Epub 2022 Aug 18.
Early introduction and distributed learning have been shown to improve student comfort with basic requisite suturing skills. The need for more frequent and directed feedback, however, remains an enduring concern for both remote and in-person training. A previous in-person curriculum for our second-year medical students transitioning to clerkships was adapted to an at-home video-based assessment model due to the social distancing implications of COVID-19. We aimed to develop an Artificial Intelligence (AI) model to perform video-based assessment.
Second-year medical students were asked to submit a video of a simple interrupted knot on a penrose drain with instrument tying technique after self-training to proficiency. Proficiency was defined as performing the task under two minutes with no critical errors. All the videos were first manually rated with a pass-fail rating and then subsequently underwent task segmentation. We developed and trained two AI models based on convolutional neural networks to identify errors (instrument holding and knot-tying) and provide automated ratings.
A total of 229 medical student videos were reviewed (150 pass, 79 fail). Of those who failed, the critical error distribution was 15 knot-tying, 47 instrument-holding, and 17 multiple. A total of 216 videos were used to train the models after excluding the low-quality videos. A k-fold cross-validation (k = 10) was used. The accuracy of the instrument holding model was 89% with an F-1 score of 74%. For the knot-tying model, the accuracy was 91% with an F-1 score of 54%.
Medical students require assessment and directed feedback to better acquire surgical skill, but this is often time-consuming and inadequately done. AI techniques can instead be employed to perform automated surgical video analysis. Future work will optimize the current model to identify discrete errors in order to supplement video-based rating with specific feedback.
早期介绍和分布式学习已被证明可以提高学生对基本必要缝合技能的舒适度。然而，无论是远程培训还是现场培训，都需要更频繁和更有针对性的反馈，这仍然是一个持久的问题。由于 COVID-19 的社交距离影响，我们为即将进入实习阶段的二年级医学生设计的现场课程被改编为基于家庭视频的评估模式。我们旨在开发一种人工智能 (AI) 模型来进行视频评估。
二年级医学生在自我训练达到熟练程度后，被要求提交一段在 Penrose 引流管上用器械打结技术进行简单间断结的视频。熟练程度定义为在两分钟内完成任务，且无关键错误。所有视频首先进行手动评分，通过/不通过评分，然后进行任务分割。我们开发并训练了两个基于卷积神经网络的 AI 模型来识别错误（器械握持和打结）并提供自动评分。
共审查了 229 名医学生的视频（150 个通过，79 个失败）。在失败的学生中，关键错误分布为 15 个打结错误、47 个器械握持错误和 17 个多个错误。在排除低质量视频后，共使用了 216 个视频来训练模型。使用 k 折交叉验证（k=10）。器械握持模型的准确率为 89%，F1 得分为 74%。对于打结模型，准确率为 91%，F1 得分为 54%。
医学生需要评估和有针对性的反馈，以更好地掌握手术技能，但这通常很耗时且做得不够充分。人工智能技术可以代替人工进行自动手术视频分析。未来的工作将优化当前模型，以识别离散错误，以便在基于视频的评分中提供更具体的反馈。
