Department of Radiology, Massachusetts General Hospital, Boston, MA, United States; School of Engineering and Applied Sciences, Harvard University, Boston, MA, United States.
Department of Radiology, Massachusetts General Hospital, Boston, MA, United States.
Med Image Anal. 2021 May;70:101993. doi: 10.1016/j.media.2021.101993. Epub 2021 Feb 7.
In recent years, deep learning-based image analysis methods have been widely applied in computer-aided detection, diagnosis and prognosis, and has shown its value during the public health crisis of the novel coronavirus disease 2019 (COVID-19) pandemic. Chest radiograph (CXR) has been playing a crucial role in COVID-19 patient triaging, diagnosing and monitoring, particularly in the United States. Considering the mixed and unspecific signals in CXR, an image retrieval model of CXR that provides both similar images and associated clinical information can be more clinically meaningful than a direct image diagnostic model. In this work we develop a novel CXR image retrieval model based on deep metric learning. Unlike traditional diagnostic models which aim at learning the direct mapping from images to labels, the proposed model aims at learning the optimized embedding space of images, where images with the same labels and similar contents are pulled together. The proposed model utilizes multi-similarity loss with hard-mining sampling strategy and attention mechanism to learn the optimized embedding space, and provides similar images, the visualizations of disease-related attention maps and useful clinical information to assist clinical decisions. The model is trained and validated on an international multi-site COVID-19 dataset collected from 3 different sources. Experimental results of COVID-19 image retrieval and diagnosis tasks show that the proposed model can serve as a robust solution for CXR analysis and patient management for COVID-19. The model is also tested on its transferability on a different clinical decision support task for COVID-19, where the pre-trained model is applied to extract image features from a new dataset without any further training. The extracted features are then combined with COVID-19 patient's vitals, lab tests and medical histories to predict the possibility of airway intubation in 72 hours, which is strongly associated with patient prognosis, and is crucial for patient care and hospital resource planning. These results demonstrate our deep metric learning based image retrieval model is highly efficient in the CXR retrieval, diagnosis and prognosis, and thus has great clinical value for the treatment and management of COVID-19 patients.
近年来，基于深度学习的图像分析方法已广泛应用于计算机辅助检测、诊断和预后，在新型冠状病毒病 2019（COVID-19）大流行这一公共卫生危机期间显示出了其价值。胸部 X 线摄影（CXR）在 COVID-19 患者分诊、诊断和监测中发挥了至关重要的作用，尤其是在美国。考虑到 CXR 中混合且非特异性的信号，一个提供相似图像和相关临床信息的 CXR 图像检索模型可能比直接的图像诊断模型更具有临床意义。在这项工作中，我们开发了一种新的基于深度度量学习的 CXR 图像检索模型。与旨在学习从图像到标签的直接映射的传统诊断模型不同，所提出的模型旨在学习图像的优化嵌入空间，其中具有相同标签和相似内容的图像被聚集在一起。所提出的模型利用多相似性损失和硬挖掘采样策略以及注意力机制来学习优化的嵌入空间，并提供相似的图像、疾病相关注意力图的可视化以及有用的临床信息，以辅助临床决策。该模型在来自 3 个不同来源的国际多站点 COVID-19 数据集上进行了训练和验证。COVID-19 图像检索和诊断任务的实验结果表明，所提出的模型可以作为 COVID-19 中 CXR 分析和患者管理的稳健解决方案。该模型还在 COVID-19 的另一个临床决策支持任务的可转移性上进行了测试，其中无需进一步训练就将预训练模型应用于从新数据集提取图像特征。然后将提取的特征与 COVID-19 患者的生命体征、实验室检查和病史相结合，以预测 72 小时内气道插管的可能性，这与患者预后密切相关，对患者护理和医院资源规划至关重要。这些结果表明，我们基于深度度量学习的图像检索模型在 CXR 检索、诊断和预后方面非常高效，因此对 COVID-19 患者的治疗和管理具有重要的临床价值。
