Department of Cardiac Surgery, Heidelberg University Hospital, Heidelberg, Germany.
DZHK (German Centre for Cardiovascular Research), Partner Site Heidelberg/Mannheim, Heidelberg, Germany.
Int J Comput Assist Radiol Surg. 2024 Apr;19(4):699-711. doi: 10.1007/s11548-024-03063-9. Epub 2024 Jan 29.
Machine learning approaches can only be reliably evaluated if training, validation, and test data splits are representative and not affected by the absence of classes. Surgical workflow and instrument recognition are two tasks that are complicated in this manner, because of heavy data imbalances resulting from different length of phases and their potential erratic occurrences. Furthermore, sub-properties like instrument (co-)occurrence are usually not particularly considered when defining the split.
We present a publicly available data visualization tool that enables interactive exploration of dataset partitions for surgical phase and instrument recognition. The application focuses on the visualization of the occurrence of phases, phase transitions, instruments, and instrument combinations across sets. Particularly, it facilitates assessment of dataset splits, especially regarding identification of sub-optimal dataset splits.
We performed analysis of the datasets Cholec80, CATARACTS, CaDIS, M2CAI-workflow, and M2CAI-tool using the proposed application. We were able to uncover phase transitions, individual instruments, and combinations of surgical instruments that were not represented in one of the sets. Addressing these issues, we identify possible improvements in the splits using our tool. A user study with ten participants demonstrated that the participants were able to successfully solve a selection of data exploration tasks.
In highly unbalanced class distributions, special care should be taken with respect to the selection of an appropriate dataset split because it can greatly influence the assessments of machine learning approaches. Our interactive tool allows for determination of better splits to improve current practices in the field. The live application is available at https://cardio-ai.github.io/endovis-ml/ .
如果训练、验证和测试数据的划分具有代表性，并且不受类缺失的影响，那么机器学习方法才能得到可靠的评估。由于不同阶段长度的严重数据不平衡以及它们潜在的不规则发生，手术流程和器械识别是两种以这种方式变得复杂的任务。此外，在定义划分时，通常不会特别考虑像器械（共同）出现这样的子属性。
我们提出了一个公开可用的数据可视化工具，该工具支持交互式探索手术阶段和器械识别的数据集划分。该应用程序侧重于跨集显示阶段、阶段转换、器械和器械组合的发生。特别是，它便于评估数据集划分，特别是识别次优数据集划分。
我们使用提出的应用程序对数据集 Cholec80、CATARACTS、CaDIS、M2CAI-workflow 和 M2CAI-tool 进行了分析。我们能够发现一个集中没有表示的阶段转换、单个器械和手术器械的组合。解决这些问题，我们可以使用我们的工具识别出可能改进划分的方法。一项由 10 名参与者进行的用户研究表明，参与者能够成功地完成一系列数据探索任务。
在高度不平衡的类分布中，应特别注意选择适当的数据集划分，因为它会极大地影响机器学习方法的评估。我们的交互式工具允许确定更好的划分，以改进该领域的当前实践。实时应用程序可在 https://cardio-ai.github.io/endovis-ml/ 上获得。
