Medtronic Digital Surgery, 230 City Rd, EC1V 2QY, London, United Kingdom; Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London (UCL), 43-45 Foley St, W1W 7TY, London, United Kingdom.
Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London (UCL), 43-45 Foley St, W1W 7TY, London, United Kingdom.
Med Image Anal. 2024 Oct;97:103263. doi: 10.1016/j.media.2024.103263. Epub 2024 Jul 3.
The lack of large datasets and high-quality annotated data often limits the development of accurate and robust machine-learning models within the medical and surgical domains. In the machine learning community, generative models have recently demonstrated that it is possible to produce novel and diverse synthetic images that closely resemble reality while controlling their content with various types of annotations. However, generative models have not been yet fully explored in the surgical domain, partially due to the lack of large datasets and due to specific challenges present in the surgical domain such as the large anatomical diversity. We propose Surgery-GAN, a novel generative model that produces synthetic images from segmentation maps. Our architecture produces surgical images with improved quality when compared to early generative models thanks to the combination of channel- and pixel-level normalization layers that boost image quality while granting adherence to the input segmentation map. While state-of-the-art generative models often generate overfitted images, lacking diversity, or containing unrealistic artefacts such as cartooning; experiments demonstrate that Surgery-GAN is able to generate novel, realistic, and diverse surgical images in three different surgical datasets: cholecystectomy, partial nephrectomy, and radical prostatectomy. In addition, we investigate whether the use of synthetic images together with real ones can be used to improve the performance of other machine-learning models. Specifically, we use Surgery-GAN to generate large synthetic datasets which we then use to train five different segmentation models. Results demonstrate that using our synthetic images always improves the mean segmentation performance with respect to only using real images. For example, when considering radical prostatectomy, we can boost the mean segmentation performance by up to 5.43%. More interestingly, experimental results indicate that the performance improvement is larger in the set of classes that are under-represented in the training sets, where the performance boost of specific classes reaches up to 61.6%.
缺乏大型数据集和高质量标注数据通常限制了医学和外科领域内准确和稳健的机器学习模型的发展。在机器学习社区中，生成模型最近证明，通过使用各种类型的标注来控制其内容，生成新颖且多样化的、与现实紧密相似的合成图像是可能的。然而，生成模型在外科领域尚未得到充分探索，部分原因是缺乏大型数据集，以及外科领域存在特定的挑战，例如解剖结构多样性大。我们提出了 Surgery-GAN，这是一种从分割图生成合成图像的新型生成模型。与早期的生成模型相比，我们的架构通过结合通道级和像素级归一化层来生成具有更高质量的手术图像，同时保证对输入分割图的遵循。虽然最先进的生成模型通常会生成过度拟合的图像，缺乏多样性，或包含不真实的伪影，如卡通化；但实验表明，Surgery-GAN 能够在三个不同的外科数据集（胆囊切除术、部分肾切除术和根治性前列腺切除术）中生成新颖、真实和多样化的手术图像。此外，我们还研究了是否可以将合成图像与真实图像一起使用来提高其他机器学习模型的性能。具体来说，我们使用 Surgery-GAN 生成大型合成数据集，然后使用这些数据集来训练五个不同的分割模型。结果表明，使用我们的合成图像总是可以提高仅使用真实图像的平均分割性能。例如，在考虑根治性前列腺切除术时，我们可以将平均分割性能提高高达 5.43%。更有趣的是，实验结果表明，在训练集中代表性不足的类别的性能提升更大，特定类别的性能提升高达 61.6%。
