Haq Imran Ul, Mhamed Mustafa, Al-Harbi Mohammed, Osman Hamid, Hamd Zuhal Y, Liu Zhe
School of Computer Science and Communication Engineering, Jiangsu University, Zhenjiang 212013, China.
College of Information and Electrical Engineering, China Agriculture University, Beijing 100083, China.
Bioengineering (Basel). 2025 Apr 30;12(5):477. doi: 10.3390/bioengineering12050477.
The majority of data collected and obtained from various sources over a patient's lifetime can be assumed to comprise pertinent information for delivering the best possible treatment. Medical data, such as radiographic and histopathology images, electrocardiograms, and medical records, all guide a physician's diagnostic approach. Nevertheless, most machine learning techniques in the healthcare field emphasize data analysis from a single modality, which is insufficiently reliable. This is especially evident in radiology, which has long been an essential topic of machine learning in healthcare because of its high data density, availability, and interpretation capability. In the future, computer-assisted diagnostic systems must be intelligent to process a variety of data simultaneously, similar to how doctors examine various resources while diagnosing patients. By extracting novel characteristics from diverse medical data sources, advanced identification techniques known as multimodal learning may be applied, enabling algorithms to analyze data from various sources and eliminating the need to train each modality. This approach enhances the flexibility of algorithms by incorporating diverse data. A growing quantity of current research has focused on the exploration of extracting data from multiple sources and constructing precise multimodal machine/deep learning models for medical examinations. A comprehensive analysis and synthesis of recent publications focusing on multimodal machine learning in detecting diseases is provided. Potential future research directions are also identified. This review presents an overview of multimodal machine learning (MMML) in radiology, a field at the cutting edge of integrating artificial intelligence into medical imaging. As radiological practices continue to evolve, the combination of various imaging and non-imaging data modalities is gaining increasing significance. This paper analyzes current methodologies, applications, and trends in MMML while outlining challenges and predicting upcoming research directions. Beginning with an overview of the different data modalities involved in radiology, namely, imaging, text, and structured medical data, this review explains the processes of modality fusion, representation learning, and modality translation, showing how they boost diagnosis efficacy and improve patient care. Additionally, this review discusses key datasets that have been instrumental in advancing MMML research. This review may help clinicians and researchers comprehend the spatial distribution of the field, outline the current level of advancement, and identify areas of research that need to be explored regarding MMML in radiology.
可以假定,在患者一生中从各种来源收集和获取的大多数数据都包含有助于提供最佳治疗的相关信息。医学数据,如放射影像和组织病理学图像、心电图以及病历,都为医生的诊断方法提供指导。然而,医疗保健领域的大多数机器学习技术都强调对单一模态的数据分析,其可靠性不足。这在放射学中尤为明显,由于其高数据密度、可用性和解读能力,放射学长期以来一直是医疗保健领域机器学习的一个重要主题。未来,计算机辅助诊断系统必须智能化,以便能够同时处理各种数据,就像医生在诊断患者时检查各种资源一样。通过从不同的医学数据源中提取新特征,可以应用称为多模态学习的先进识别技术,使算法能够分析来自各种来源的数据,而无需对每种模态进行单独训练。这种方法通过纳入多样化的数据提高了算法的灵活性。当前越来越多的研究致力于探索从多个来源提取数据,并构建用于医学检查的精确多模态机器/深度学习模型。本文对近期专注于疾病检测中的多模态机器学习的出版物进行了全面的分析和综合。同时还确定了潜在的未来研究方向。本综述概述了放射学中的多模态机器学习(MMML),这是一个将人工智能整合到医学成像领域的前沿领域。随着放射学实践的不断发展,各种成像和非成像数据模态的结合正变得越来越重要。本文分析了MMML的当前方法、应用和趋势,同时概述了挑战并预测了未来的研究方向。本综述首先概述了放射学中涉及的不同数据模态,即成像、文本和结构化医学数据,然后解释了模态融合、表征学习和模态转换的过程,展示了它们如何提高诊断效率并改善患者护理。此外,本综述还讨论了对推动MMML研究起到重要作用的关键数据集。本综述可能有助于临床医生和研究人员了解该领域的空间分布,概述当前的进展水平,并确定放射学中关于MMML需要探索的研究领域。