Graduate School of Science and Technology, Keio University, Yokohama, 2238522, Japan.
Institute of Systems and Information Engineering, University of Tsukuba, Tsukuba, 3058573, Japan.
Int J Comput Assist Radiol Surg. 2024 Nov;19(11):2195-2202. doi: 10.1007/s11548-024-03074-6. Epub 2024 Feb 27.
Analysis of operative fields is expected to aid in estimating procedural workflow and evaluating surgeons' procedural skills by considering the temporal transitions during the progression of the surgery. This study aims to propose an automatic recognition system for the procedural workflow by employing machine learning techniques to identify and distinguish elements in the operative field, including body tissues such as fat, muscle, and dermis, along with surgical tools.
We conducted annotations on approximately 908 first-person-view images of breast surgery to facilitate segmentation. The annotated images were used to train a pixel-level classifier based on Mask R-CNN. To assess the impact on procedural workflow recognition, we annotated an additional 43,007 images. The network, structured on the Transformer architecture, was then trained with surgical images incorporating masks for body tissues and surgical tools.
The instance segmentation of each body tissue in the segmentation phase provided insights into the trend of area transitions for each tissue. Simultaneously, the spatial features of the surgical tools were effectively captured. In regard to the accuracy of procedural workflow recognition, accounting for body tissues led to an average improvement of 3 % over the baseline. Furthermore, the inclusion of surgical tools yielded an additional increase in accuracy by 4 % compared to the baseline.
In this study, we revealed the contribution of the temporal transition of the body tissues and surgical tools spatial features to recognize procedural workflow in first-person-view surgical videos. Body tissues, especially in open surgery, can be a crucial element. This study suggests that further improvements can be achieved by accurately identifying surgical tools specific to each procedural workflow step.
通过考虑手术进展过程中的时间转换，对手术视野进行分析，有望帮助估计手术流程并评估外科医生的手术技能。本研究旨在通过采用机器学习技术来识别和区分手术视野中的元素，包括脂肪、肌肉和真皮等身体组织以及手术工具，从而提出一种手术流程的自动识别系统。
我们对大约 908 张第一人称视角的乳房手术图像进行了注释，以方便分割。使用标注图像基于 Mask R-CNN 训练了一个像素级分类器。为了评估对手术流程识别的影响，我们又标注了 43007 张图像。然后，使用包含身体组织和手术工具掩模的手术图像对基于 Transformer 架构的网络进行了训练。
分割阶段中对每个身体组织的实例分割提供了每个组织区域过渡趋势的见解。同时，有效地捕获了手术工具的空间特征。在手术流程识别的准确性方面，考虑身体组织的情况下，平均比基线提高了 3%。此外，与基线相比，包含手术工具的情况下，准确性又额外提高了 4%。
在本研究中，我们揭示了身体组织的时间过渡和手术工具空间特征的时间过渡对识别第一人称视角手术视频中的手术流程的贡献。身体组织，尤其是在开放式手术中，可能是一个关键因素。本研究表明，通过准确识别每个手术流程步骤特有的手术工具，可以进一步提高识别的准确性。
