Department of Biomedical Engineering, Tel-Aviv University, Tel Aviv, Israel.
Department of Diagnostic Imaging, Sheba Medical Center, Ramat Gan, Israel affiliated with the Tel Aviv University, Tel Aviv, Israel.
Sci Rep. 2023 May 9;13(1):7544. doi: 10.1038/s41598-023-34303-8.
Pulmonary embolism (PE) is a common, life threatening cardiovascular emergency. Risk stratification is one of the core principles of acute PE management and determines the choice of diagnostic and therapeutic strategies. In routine clinical practice, clinicians rely on the patient's electronic health record (EHR) to provide a context for their medical imaging interpretation. Most deep learning models for radiology applications only consider pixel-value information without the clinical context. Only a few integrate both clinical and imaging data. In this work, we develop and compare multimodal fusion models that can utilize multimodal data by combining both volumetric pixel data and clinical patient data for automatic risk stratification of PE. Our best performing model is an intermediate fusion model that incorporates both bilinear attention and TabNet, and can be trained in an end-to-end manner. The results show that multimodality boosts performance by up to 14% with an area under the curve (AUC) of 0.96 for assessing PE severity, with a sensitivity of 90% and specificity of 94%, thus pointing to the value of using multimodal data to automatically assess PE severity.
肺栓塞（PE）是一种常见的、危及生命的心血管急症。风险分层是急性 PE 管理的核心原则之一，决定了诊断和治疗策略的选择。在常规临床实践中，临床医生依赖患者的电子健康记录（EHR）为其医学影像解释提供背景信息。大多数用于放射学应用的深度学习模型仅考虑像素值信息，而不考虑临床背景。只有少数模型同时整合了临床和影像数据。在这项工作中，我们开发并比较了多模态融合模型，这些模型可以通过结合容积像素数据和临床患者数据来利用多模态数据，从而实现 PE 的自动风险分层。我们表现最好的模型是一种中间融合模型，它结合了双线性注意力和 TabNet，可以端到端地进行训练。结果表明，多模态可将性能提高多达 14%，PE 严重程度评估的曲线下面积（AUC）为 0.96，灵敏度为 90%，特异性为 94%，这表明使用多模态数据自动评估 PE 严重程度的价值。
