University of California, San Francisco-East Bay, General Surgery, Oakland, CA, USA.
Johnson & Johnson MedTech, New Brunswick, NJ, USA.
Surg Endosc. 2023 Jul;37(7):5665-5672. doi: 10.1007/s00464-023-09870-6. Epub 2023 Jan 19.
Artificial intelligence (AI) can automate certain tasks to improve data collection. Models have been created to annotate the steps of Roux-en-Y Gastric Bypass (RYGB). However, model performance has not been compared with individual surgeon annotator performance. We developed a model that automatically labels RYGB steps and compares its performance to surgeons.
545 videos (17 surgeons) of laparoscopic RYGB procedures were collected. An annotation guide (12 steps, 52 tasks) was developed. Steps were annotated by 11 surgeons. Each video was annotated by two surgeons and a third reconciled the differences. A convolutional AI model was trained to identify steps and compared with manual annotation. For modeling, we used 390 videos for training, 95 for validation, and 60 for testing. The performance comparison between AI model versus manual annotation was performed using ANOVA (Analysis of Variance) in a subset of 60 testing videos. We assessed the performance of the model at each step and poor performance was defined (F1-score < 80%).
The convolutional model identified 12 steps in the RYGB architecture. Model performance varied at each step [F1 > 90% for 7, and > 80% for 2]. The reconciled manual annotation data (F1 > 80% for > 5 steps) performed better than trainee's (F1 > 80% for 2-5 steps for 4 annotators, and < 2 steps for 4 annotators). In testing subset, certain steps had low performance, indicating potential ambiguities in surgical landmarks. Additionally, some videos were easier to annotate than others, suggesting variability. After controlling for variability, the AI algorithm was comparable to the manual (p < 0.0001).
AI can be used to identify surgical landmarks in RYGB comparable to the manual process. AI was more accurate to recognize some landmarks more accurately than surgeons. This technology has the potential to improve surgical training by assessing the learning curves of surgeons at scale.
人工智能 (AI) 可以自动化某些任务，以提高数据收集效率。已经创建了模型来注释 Roux-en-Y 胃旁路术 (RYGB) 的步骤。然而，模型性能尚未与个别外科医生的注释性能进行比较。我们开发了一种自动标记 RYGB 步骤并比较其性能与外科医生的模型。
收集了 545 个腹腔镜 RYGB 手术视频（17 位外科医生）。制定了一个注释指南（12 个步骤，52 个任务）。由 11 位外科医生注释步骤。每个视频都由两位外科医生进行注释，第三位则对差异进行协调。训练了一个卷积 AI 模型来识别步骤，并与手动注释进行比较。在建模中，我们使用 390 个视频进行训练，95 个用于验证，60 个用于测试。在 60 个测试视频的子集上，使用方差分析 (ANOVA) 对 AI 模型与手动注释之间的性能进行了比较。我们评估了模型在每个步骤中的性能，将性能差的定义为（F1 分数 < 80%）。
卷积模型识别了 RYGB 结构中的 12 个步骤。模型性能在每个步骤上有所不同[F1>90% 的 7 个，F1>80% 的 2 个]。经过协调的手动注释数据（F1>80% 的 5 个以上步骤）比学员的表现更好（F1>80% 的 4 个注释者为 2-5 个步骤，而 4 个注释者为 <2 个步骤）。在测试子集中，某些步骤的性能较低，表明手术标志的潜在歧义。此外，一些视频比其他视频更容易注释，这表明存在变异性。在控制变异性后，AI 算法与手动算法相当（p<0.0001）。
AI 可用于识别 RYGB 中的手术标志，与手动过程相当。AI 比外科医生更准确地识别某些标志。这项技术有可能通过评估外科医生的学习曲线来提高手术培训的规模。
