Zhong Bin, Zhang Runan, Luo Shuixiang, Zheng Jie
Department of Respiratory Medicine, The First Affiliated Hospital of Gannan Medical University, Ganzhou, Jiangxi, China.
College of Pharmacy, Gannan Medical University, Ganzhou, Jiangxi, China.
Front Med (Lausanne). 2024 Nov 18;11:1446936. doi: 10.3389/fmed.2024.1446936. eCollection 2024.
This study aims to address the potential and challenges of multimodal medical information in the diagnosis of interstitial lung disease (ILD) by developing an ILD identification model (ILDIM) based on the multimodal fusion attention mechanism (MFAM) to improve the accuracy and reliability of ILD. Large-scale multimodal medical information data, including chest CT image slices, physiological indicator time series data, and patient history text information were collected. These data are professionally cleaned and normalized to ensure data quality and consistency. Convolutional Neural Network (CNN) is used to extract CT image features, Bidirectional Long Short-Term Memory Network (Bi-LSTM) model is used to learn temporal physiological metrics data under long-term dependency, and Self-Attention Mechanism is used to encode textual semantic information in patient's self-reporting and medical prescriptions. In addition, the multimodal perception mechanism uses a Transformer-based model to improve the diagnostic performance of ILD by learning the importance weights of each modality's data to optimally fuse the different modalities. Finally, the ablation test and comparison results show that the model performs well in terms of comprehensive performance. By combining multimodal data sources, the model not only improved the Precision, Recall and F1 score, but also significantly increased the AUC value. This suggests that the combined use of different modal information can provide a more comprehensive assessment of a patient's health status, thereby improving the diagnostic comprehensiveness and accuracy of ILD. This study also considered the computational complexity of the model, and the results show that ILDIM-MFAM has a relatively low number of model parameters and computational complexity, which is very favorable for practical deployment and operational efficiency.
本研究旨在通过开发基于多模态融合注意力机制（MFAM）的间质性肺疾病（ILD）识别模型（ILDIM）来解决多模态医学信息在ILD诊断中的潜力和挑战，以提高ILD诊断的准确性和可靠性。收集了大规模的多模态医学信息数据，包括胸部CT图像切片、生理指标时间序列数据和患者病史文本信息。这些数据经过专业清理和归一化处理，以确保数据质量和一致性。使用卷积神经网络（CNN）提取CT图像特征，使用双向长短期记忆网络（Bi-LSTM）模型学习长期依赖下的时间生理指标数据，并使用自注意力机制对患者自述和医学处方中的文本语义信息进行编码。此外，多模态感知机制使用基于Transformer的模型，通过学习各模态数据的重要性权重来优化融合不同模态，从而提高ILD的诊断性能。最后，消融测试和比较结果表明，该模型在综合性能方面表现良好。通过结合多模态数据源，该模型不仅提高了精确率、召回率和F1分数，还显著提高了AUC值。这表明不同模态信息的联合使用可以更全面地评估患者的健康状况，从而提高ILD诊断的全面性和准确性。本研究还考虑了模型的计算复杂度，结果表明ILDIM-MFAM的模型参数数量和计算复杂度相对较低，这对实际部署和运行效率非常有利。
