Niu Ke, Zhang Ke, Peng Xueping, Pan Yijie, Xiao Naian
Computer School, Beijing Information Science and Technology University, Beijing, China.
Australian Artificial Intelligence Institute, Faculty of Engineering and Information Technology, University of Technology Sydney, Sydney, NSW, Australia.
Front Mol Biosci. 2023 Mar 8;10:1136071. doi: 10.3389/fmolb.2023.1136071. eCollection 2023.
In intensive care units (ICUs), mortality prediction is performed by combining information from these two sources of ICU patients by monitoring patient health. Respectively, time series data generated from each patient admission to the ICU and clinical records consisting of physician diagnostic summaries. However, existing mortality prediction studies mainly cascade the multimodal features of time series data and clinical records for prediction, ignoring thecross-modal correlation between the underlying features in different modal data. To address theseissues, we propose a multimodal fusion model for mortality prediction that jointly models patients' time-series data as well as clinical records. We apply a fine-tuned Bert model (Bio-Bert) to the patient's clinical record to generate a holistic embedding of the text part, which is then combined with the output of an LSTM model encoding the patient's time-series data to extract valid features. The global contextual information of each modal data is extracted using the improved fusion module to capture the correlation between different modal data. Furthermore, the improved fusion module can be easily added to the fusion features of any unimodal network and utilize existing pre-trained unimodal model weights. We use a real dataset containing 18904 ICU patients to train and evaluate our model, and the research results show that the representations obtained by themodel can achieve better prediction accuracy compared to the baseline.
在重症监护病房（ICU）中，死亡率预测是通过监测患者健康状况，将来自这两种ICU患者数据源的信息相结合来进行的。具体而言，是由每位患者入住ICU时生成的时间序列数据以及由医生诊断摘要组成的临床记录。然而，现有的死亡率预测研究主要是将时间序列数据和临床记录的多模态特征串联起来进行预测，忽略了不同模态数据中潜在特征之间的跨模态相关性。为了解决这些问题，我们提出了一种用于死亡率预测的多模态融合模型，该模型联合对患者的时间序列数据以及临床记录进行建模。我们将一个微调后的Bert模型（Bio-Bert）应用于患者的临床记录，以生成文本部分的整体嵌入，然后将其与对患者时间序列数据进行编码的LSTM模型的输出相结合，以提取有效特征。使用改进的融合模块提取每个模态数据的全局上下文信息，以捕捉不同模态数据之间的相关性。此外，改进的融合模块可以轻松添加到任何单模态网络的融合特征中，并利用现有的预训练单模态模型权重。我们使用一个包含18904名ICU患者的真实数据集来训练和评估我们的模型，研究结果表明，与基线相比，该模型获得的表征能够实现更好的预测准确性。
