Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA.
Laboratory for Computational Sensing + Robotics, Johns Hopkins University, Baltimore, MD, USA.
Int J Comput Assist Radiol Surg. 2019 Sep;14(9):1517-1528. doi: 10.1007/s11548-019-02011-2. Epub 2019 Jun 11.
Machine learning-based approaches now outperform competing methods in most disciplines relevant to diagnostic radiology. Image-guided procedures, however, have not yet benefited substantially from the advent of deep learning, in particular because images for procedural guidance are not archived and thus unavailable for learning, and even if they were available, annotations would be a severe challenge due to the vast amounts of data. In silico simulation of X-ray images from 3D CT is an interesting alternative to using true clinical radiographs since labeling is comparably easy and potentially readily available.
We extend our framework for fast and realistic simulation of fluoroscopy from high-resolution CT, called DeepDRR, with tool modeling capabilities. The framework is publicly available, open source, and tightly integrated with the software platforms native to deep learning, i.e., Python, PyTorch, and PyCuda. DeepDRR relies on machine learning for material decomposition and scatter estimation in 3D and 2D, respectively, but uses analytic forward projection and noise injection to ensure acceptable computation times. On two X-ray image analysis tasks, namely (1) anatomical landmark detection and (2) segmentation and localization of robot end-effectors, we demonstrate that convolutional neural networks (ConvNets) trained on DeepDRRs generalize well to real data without re-training or domain adaptation. To this end, we use the exact same training protocol to train ConvNets on naïve and DeepDRRs and compare their performance on data of cadaveric specimens acquired using a clinical C-arm X-ray system.
Our findings are consistent across both considered tasks. All ConvNets performed similarly well when evaluated on the respective synthetic testing set. However, when applied to real radiographs of cadaveric anatomy, ConvNets trained on DeepDRRs significantly outperformed ConvNets trained on naïve DRRs ([Formula: see text]).
Our findings for both tasks are positive and promising. Combined with complementary approaches, such as image style transfer, the proposed framework for fast and realistic simulation of fluoroscopy from CT contributes to promoting the implementation of machine learning in X-ray-guided procedures. This paradigm shift has the potential to revolutionize intra-operative image analysis to simplify surgical workflows.
基于机器学习的方法在与诊断放射学相关的大多数领域都已超越竞争方法。然而，图像引导程序尚未从深度学习的出现中显著受益，特别是因为程序引导用图像未被存档，因此无法用于学习，即使可用，由于数据量庞大，注释也将是一个严峻的挑战。从 3D CT 模拟 X 射线图像是使用真实临床射线照片的一种有趣替代方法，因为标记相对容易且可能随时可用。
我们通过工具建模功能扩展了我们用于从高分辨率 CT 快速逼真地模拟透视的框架，称为 DeepDRR。该框架是公开的、开源的，并且与深度学习的本机软件平台紧密集成，即 Python、PyTorch 和 PyCuda。DeepDRR 分别依赖机器学习进行 3D 和 2D 中的材料分解和散射估计，但使用分析前向投影和噪声注入来确保可接受的计算时间。在两个 X 射线图像分析任务上，即（1）解剖学地标检测和（2）机器人末端执行器的分割和定位，我们证明了在无需重新训练或领域适应的情况下，在 DeepDRR 上训练的卷积神经网络（ConvNets）可以很好地推广到真实数据。为此，我们使用完全相同的训练协议在天真和 DeepDRR 上训练 ConvNets，并比较它们在使用临床 C 臂 X 射线系统获取的尸体标本数据上的性能。
我们的发现对于两个任务都是一致的。所有 ConvNets 在各自的合成测试集上的评估结果都相似。然而，当应用于尸体解剖学的真实射线照片时，在 DeepDRR 上训练的 ConvNets明显优于在天真 DRR 上训练的 ConvNets（[公式：见正文]）。
我们对两个任务的发现都是积极和有希望的。与互补方法（如图像样式转换）结合使用，用于从 CT 快速逼真地模拟透视的建议框架有助于推动机器学习在 X 射线引导程序中的实施。这种范式转变有可能简化手术工作流程，从而彻底改变术中图像分析。
