Bender Johannes, Kwe Jeremy, Hoeh Benedikt, Boehm Katharina, Platzek Ivan, Borkowetz Angelika, Speidel Stefanie, Pfeiffer Micha
Department of Translational Surgical Oncology, NCT/UCC Dresden, Dresden, Germany.
German Cancer Research Center (DKFZ), Heidelberg, Germany.
Int J Comput Assist Radiol Surg. 2025 Jun;20(6):1265-1272. doi: 10.1007/s11548-025-03374-5. Epub 2025 May 8.
Markerless navigation in minimally invasive surgery is still an unsolved challenge. Many proposed navigation systems for minimally invasive surgeries rely on stereoscopic images, while in clinical practice oftentimes monocular endoscopes are used. Combined with the lack of automatic video-based navigation systems for prostatectomies, this paper explores methods to tackle both research gaps at the same time for robot-assisted prostatectomies.
In order to realize a semi-automatic augmented reality overlay for navigated prostatectomy, the camera pose w.r.t. the prostate needs to be estimated. We developed a method where visual cues are drawn on top of the organ after an initial manual alignment, simultaneously creating matching landmarks on the 2D and 3D data. Starting from this key frame, the cues are then tracked in the endoscopic video. Both PnPRansac and differentiable rendering are then explored to perform 2D-3D registration for each frame.
We performed experiments on synthetic and in vivo data. On synthetic data differentiable rendering can achieve a median target registration error of 6.11 mm. Both PnPRansac and differentiable rendering are feasible methods for 2D-3D registration.
We demonstrated a video-based markerless augmented reality overlay for navigated prostatectomy, using visual cues as an anchor.
微创手术中的无标记导航仍是一个未解决的挑战。许多提出的微创手术导航系统依赖于立体图像，而在临床实践中通常使用单目内窥镜。鉴于缺乏用于前列腺切除术的基于视频的自动导航系统，本文探索了同时解决这两个研究空白的方法，用于机器人辅助前列腺切除术。
为了实现用于导航前列腺切除术的半自动增强现实叠加，需要估计相对于前列腺的相机位姿。我们开发了一种方法，在初始手动对齐后在器官顶部绘制视觉线索，同时在二维和三维数据上创建匹配的地标。从这个关键帧开始，然后在内窥镜视频中跟踪线索。然后探索使用PnPRansac和可微渲染对每一帧进行二维到三维配准。
我们在合成数据和体内数据上进行了实验。在合成数据上，可微渲染可实现的中位目标配准误差为6.11毫米。PnPRansac和可微渲染都是用于二维到三维配准的可行方法。
我们展示了一种用于导航前列腺切除术的基于视频的无标记增强现实叠加，使用视觉线索作为锚点。
