Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China and University of Chinese Academy of Sciences, Beijing 100039, China.
Department of Ultrasound, First Affiliated Hospital of Southern University of Science and Technology, Second Clinical College of Jinan University, Shenzhen People's Hospital, Shenzhen, 518020, China.
Ultrasonics. 2022 May;122:106706. doi: 10.1016/j.ultras.2022.106706. Epub 2022 Feb 7.
Accurate segmentation of kidney in ultrasound images is a vital procedure in clinical diagnosis and interventional operation. In recent years, deep learning technology has demonstrated promising prospects in medical image analysis. However, due to the inherent problems of ultrasound images, data with annotations are scarce and arduous to acquire, hampering the application of data-hungry deep learning methods. In this paper, we propose cross-modal transfer learning from computerized tomography (CT) to ultrasound (US) by leveraging annotated data in the CT modality. In particular, we adopt cycle generative adversarial network (CycleGAN) to synthesize US images from CT data and construct a transition dataset to mitigate the immense domain discrepancy between US and CT. Mainstream convolutional neural networks such as U-Net, U-Res, PSPNet, and DeepLab v3+ are pretrained on the transition dataset and then transferred to real US images. We first trained CNN models on a data set composed of 50 ultrasound images and validated them on a validation set composed of 30 ultrasound images. In addition, we selected 82 ultrasound images from another hospital to construct a cross-site data set to verify the generalization performance of the models. The experimental results show that with our proposed transfer learning strategy, the segmentation accuracy in dice similarity coefficient (DSC) reaches 0.853 for U-Net, 0.850 for U-Res, 0.826 for PSPNet and 0.827 for DeepLab v3+ on the cross-site test set. Compared with training from scratch, the accuracy improvement was 0.127, 0.097, 0.105 and 0.036 respectively. Our transfer learning strategy effectively improves the accuracy and generalization ability of ultrasound image segmentation model with limited training data.
在临床诊断和介入手术中，准确地对超声图像中的肾脏进行分割是一项至关重要的步骤。近年来，深度学习技术在医学图像分析中展现出了广阔的前景。然而，由于超声图像固有的问题，带有注释的数据稀缺且难以获取，这阻碍了数据密集型深度学习方法的应用。在本文中，我们提出了一种跨模态迁移学习方法，通过利用 CT 模态中的注释数据，将 CT 模态中的信息迁移到超声（US）模态中。具体来说，我们采用循环生成对抗网络（CycleGAN）将 CT 数据合成 US 图像，并构建一个过渡数据集，以减轻 US 和 CT 之间巨大的域差异。主流的卷积神经网络，如 U-Net、U-Res、PSPNet 和 DeepLab v3+，在过渡数据上进行预训练，然后迁移到真实的 US 图像上。我们首先在一个由 50 张超声图像组成的数据集上训练 CNN 模型，并在一个由 30 张超声图像组成的验证集上进行验证。此外，我们从另一家医院选择了 82 张超声图像来构建一个跨站点数据集，以验证模型的泛化性能。实验结果表明，在我们提出的迁移学习策略下，U-Net、U-Res、PSPNet 和 DeepLab v3+在跨站点测试集上的 Dice 相似系数（DSC）分别达到 0.853、0.850、0.826 和 0.827。与从头开始训练相比，精度分别提高了 0.127、0.097、0.105 和 0.036。我们的迁移学习策略有效地提高了有限训练数据下的超声图像分割模型的准确性和泛化能力。
