IEEE Trans Med Imaging. 2023 Oct;42(10):2924-2935. doi: 10.1109/TMI.2023.3268774. Epub 2023 Oct 2.
In recent intelligent-robot-assisted surgery studies, an urgent issue is how to detect the motion of instruments and soft tissue accurately from intra-operative images. Although optical flow technology from computer vision is a powerful solution to the motion-tracking problem, it has difficulty obtaining the pixel-wise optical flow ground truth of real surgery videos for supervised learning. Thus, unsupervised learning methods are critical. However, current unsupervised methods face the challenge of heavy occlusion in the surgical scene. This paper proposes a novel unsupervised learning framework to estimate the motion from surgical images under occlusion. The framework consists of a Motion Decoupling Network to estimate the tissue and the instrument motion with different constraints. Notably, the network integrates a segmentation subnet that estimates the segmentation map of instruments in an unsupervised manner to obtain the occlusion region and improve the dual motion estimation. Additionally, a hybrid self-supervised strategy with occlusion completion is introduced to recover realistic vision clues. Extensive experiments on two surgical datasets show that the proposed method achieves accurate motion estimation for intra-operative scenes and outperforms other unsupervised methods, with a margin of 15% in accuracy. The average estimation error for tissue is less than 2.2 pixels on average for both surgical datasets.
在最近的智能机器人辅助手术研究中，一个紧迫的问题是如何从手术图像中准确地检测器械和软组织的运动。尽管计算机视觉中的光流技术是解决运动跟踪问题的强大解决方案，但它难以获得用于监督学习的真实手术视频的逐像素光流真实值。因此，无监督学习方法至关重要。然而，目前的无监督方法面临手术场景中严重遮挡的挑战。本文提出了一种新的无监督学习框架，以在遮挡下从手术图像中估计运动。该框架由一个运动解耦网络组成，该网络使用不同的约束来估计组织和器械的运动。值得注意的是，该网络集成了一个分割子网，以无监督的方式估计器械的分割图，以获得遮挡区域并改进双运动估计。此外，还引入了一种具有遮挡完成的混合自监督策略，以恢复逼真的视觉线索。在两个手术数据集上的广泛实验表明，所提出的方法可以准确地估计手术场景中的运动，并且优于其他无监督方法，在准确性方面有 15%的优势。对于两个手术数据集，组织的平均估计误差均小于 2.2 个像素。
