Das Adrito, Khan Danyal Z, Hanrahan John G, Marcus Hani J, Stoyanov Danail
Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London, United Kingdom.
National Hospital for Neurology and Neurosurgery, University College London, United Kingdom.
Intell Based Med. 2023;8:100107. doi: 10.1016/j.ibmed.2023.100107.
Operation notes are a crucial component of patient care. However, writing them manually is prone to human error, particularly in high pressured clinical environments. Automatic generation of operation notes from video recordings can alleviate some of the administrative burdens, improve accuracy, and provide additional information. To achieve this for endoscopic pituitary surgery, 27-steps were identified via expert consensus. Then, for the 97-videos recorded for this study, a timestamp of each step was annotated by an expert surgeon. To automatically determine whether a step is present in a video, a three-stage architecture was created. Firstly, for each step, a convolution neural network was used for binary image classification on each frame of a video. Secondly, for each step, the binary frame classifications were passed to a discriminator for binary video classification. Thirdly, for each video, the binary video classifications were passed to an accumulator for multi-label step classification. The architecture was trained on 77-videos, and tested on 20-videos, where a 0.80 weighted- score was achieved. The classifications were inputted into a clinically based predefined template, and further enriched with additional video analytics. This work therefore demonstrates automatic generation of operative notes from surgical videos is feasible, and can assist surgeons during documentation.
手术记录是患者护理的关键组成部分。然而，手动书写手术记录容易出现人为错误，尤其是在压力较大的临床环境中。从视频记录中自动生成手术记录可以减轻一些管理负担、提高准确性并提供额外信息。为了在内镜垂体手术中实现这一点，通过专家共识确定了27个步骤。然后，对于本研究记录的97个视频，由一位专家外科医生标注每个步骤的时间戳。为了自动确定视频中是否存在某个步骤，创建了一个三阶段架构。首先，对于每个步骤，使用卷积神经网络对视频的每一帧进行二值图像分类。其次，对于每个步骤，将二值帧分类结果传递给一个鉴别器进行二值视频分类。第三，对于每个视频，将二值视频分类结果传递给一个累加器进行多标签步骤分类。该架构在77个视频上进行训练，并在20个视频上进行测试，加权得分达到了0.80。分类结果被输入到基于临床的预定义模板中，并通过额外的视频分析进一步丰富。因此，这项工作证明了从手术视频中自动生成手术记录是可行的，并且可以在记录过程中帮助外科医生。
