Schwimmbeck Michael, Khajarian Serouj, Auer Christopher, Wittenberg Thomas, Remmele Stefanie
Research Group Medical Technologies, University of Applied Sciences Landshut, Landshut, Germany.
Chair for Visual Computing, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany.
Int J Comput Assist Radiol Surg. 2025 Aug 5. doi: 10.1007/s11548-025-03480-4.
Augmented reality (AR) enhances surgical navigation by superimposing visible anatomical structures with three-dimensional virtual models using head-mounted displays (HMDs). In particular, interventions such as open liver surgery can benefit from AR navigation, as it aids in identifying and distinguishing tumors and risk structures. However, there is a lack of automatic and markerless methods that are robust against real-world challenges, such as partial occlusion and organ motion.
We introduce a novel multi-device approach for automatic live navigation in open liver surgery that enhances the visualization and interaction capabilities of a HoloLens 2 HMD through precise and reliable registration using an Intel RealSense RGB-D camera. The intraoperative RGB-D segmentation and the preoperative CT data are utilized to register a virtual liver model to the target anatomy. An AR-prompted Segment Anything Model (SAM) enables robust segmentation of the liver in situ without the need for additional training data. To mitigate algorithmic latency, Double Exponential Smoothing (DES) is applied to forecast registration results.
We conducted a phantom study for open liver surgery, investigating various scenarios of liver motion, viewpoints, and occlusion. The mean registration errors (8.31 mm-18.78 mm TRE) are comparable to those reported in prior work, while our approach demonstrates high success rates even for high occlusion factors and strong motion. Using forecasting, we bypassed the algorithmic latency of 79.8 ms per frame, with median forecasting errors below 2 mms and 1.5 degrees between the quaternions.
To our knowledge, this is the first work to approach markerless in situ visualization by combining a multi-device method with forecasting and a foundation model for segmentation and tracking. This enables a more reliable and precise AR registration of surgical targets with low latency. Our approach can be applied to other surgical applications and AR hardware with minimal effort.
增强现实（AR）通过头戴式显示器（HMD）将可见解剖结构与三维虚拟模型叠加，从而增强手术导航。特别是，诸如开放性肝脏手术等干预措施可受益于AR导航，因为它有助于识别和区分肿瘤及危险结构。然而，缺乏针对现实世界挑战（如部分遮挡和器官运动）具有鲁棒性的自动且无标记方法。
我们引入了一种用于开放性肝脏手术自动实时导航的新型多设备方法，该方法通过使用英特尔实感RGB-D相机进行精确且可靠的配准，增强了HoloLens 2 HMD的可视化和交互能力。术中RGB-D分割和术前CT数据用于将虚拟肝脏模型配准到目标解剖结构。一个由AR驱动的分割一切模型（SAM）能够在无需额外训练数据的情况下对肝脏进行原位鲁棒分割。为减轻算法延迟，应用双指数平滑（DES）来预测配准结果。
我们针对开放性肝脏手术进行了一项模型研究，考察了肝脏运动、视角和遮挡的各种情况。平均配准误差（TRE为8.31毫米 - 18.78毫米）与先前工作中报告的误差相当，而我们的方法即使在高遮挡因素和强烈运动的情况下也显示出高成功率。通过预测，我们绕过了每帧79.8毫秒的算法延迟，四元数之间的预测误差中位数低于2毫米和1.5度。
据我们所知，这是第一项通过将多设备方法与预测以及用于分割和跟踪的基础模型相结合来实现无标记原位可视化的工作。这使得手术目标的AR配准更可靠、精确且延迟低。我们的方法可以很容易地应用于其他手术应用和AR硬件。
