Department of Biomedical Data Science, Stanford University, Stanford, USA.
Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Stanford, USA.
Sci Rep. 2020 Dec 17;10(1):22147. doi: 10.1038/s41598-020-78888-w.
Recent advancements in deep learning have led to a resurgence of medical imaging and Electronic Medical Record (EMR) models for a variety of applications, including clinical decision support, automated workflow triage, clinical prediction and more. However, very few models have been developed to integrate both clinical and imaging data, despite that in routine practice clinicians rely on EMR to provide context in medical imaging interpretation. In this study, we developed and compared different multimodal fusion model architectures that are capable of utilizing both pixel data from volumetric Computed Tomography Pulmonary Angiography scans and clinical patient data from the EMR to automatically classify Pulmonary Embolism (PE) cases. The best performing multimodality model is a late fusion model that achieves an AUROC of 0.947 [95% CI: 0.946-0.948] on the entire held-out test set, outperforming imaging-only and EMR-only single modality models.
深度学习的最新进展使得医学成像和电子病历 (EMR) 模型在各种应用中得到了复苏，包括临床决策支持、自动化工作流程分诊、临床预测等。然而，尽管在常规实践中，临床医生依赖 EMR 为医学成像解释提供上下文，但很少有模型被开发出来以整合临床和成像数据。在这项研究中，我们开发并比较了不同的多模态融合模型架构，这些架构能够利用容积计算机断层肺动脉造影扫描的像素数据和来自 EMR 的临床患者数据，自动对肺栓塞 (PE) 病例进行分类。表现最好的多模态模型是一种后期融合模型，在整个预留测试集上的 AUROC 为 0.947[95%置信区间：0.946-0.948]，优于仅基于成像和仅基于 EMR 的单模态模型。
