Institute for Systems and Computer Engineering, Technology and Science (INESC TEC), 4200-465 Porto, Portugal.
Faculty of Engineering (FEUP), University of Porto, 4200-465 Porto, Portugal.
Sensors (Basel). 2023 Jun 15;23(12):5597. doi: 10.3390/s23125597.
In a clinical context, physicians usually take into account information from more than one data modality when making decisions regarding cancer diagnosis and treatment planning. Artificial intelligence-based methods should mimic the clinical method and take into consideration different sources of data that allow a more comprehensive analysis of the patient and, as a consequence, a more accurate diagnosis. Lung cancer evaluation, in particular, can benefit from this approach since this pathology presents high mortality rates due to its late diagnosis. However, many related works make use of a single data source, namely imaging data. Therefore, this work aims to study the prediction of lung cancer when using more than one data modality. The National Lung Screening Trial dataset that contains data from different sources, specifically, computed tomography (CT) scans and clinical data, was used for the study, the development and comparison of single-modality and multimodality models, that may explore the predictive capability of these two types of data to their full potential. A ResNet18 network was trained to classify 3D CT nodule regions of interest (ROI), whereas a random forest algorithm was used to classify the clinical data, with the former achieving an area under the ROC curve (AUC) of 0.7897 and the latter 0.5241. Regarding the multimodality approaches, three strategies, based on intermediate and late fusion, were implemented to combine the information from the 3D CT nodule ROIs and the clinical data. From those, the best model-a fully connected layer that receives as input a combination of clinical data and deep imaging features, given by a ResNet18 inference model-presented an AUC of 0.8021. Lung cancer is a complex disease, characterized by a multitude of biological and physiological phenomena and influenced by multiple factors. It is thus imperative that the models are capable of responding to that need. The results obtained showed that the combination of different types may have the potential to produce more comprehensive analyses of the disease by the models.
在临床环境中，医生在做出癌症诊断和治疗计划决策时通常会考虑来自多种数据模态的信息。基于人工智能的方法应该模仿临床方法，考虑不同的数据来源，以便更全面地分析患者，从而做出更准确的诊断。肺癌评估尤其可以受益于这种方法，因为由于诊断较晚，这种疾病的死亡率很高。然而，许多相关工作仅使用了单一的数据来源，即影像学数据。因此，本工作旨在研究使用多种数据模态进行肺癌预测。本研究使用了包含来自不同来源的数据的国家肺癌筛查试验数据集，特别是计算机断层扫描（CT）扫描和临床数据，开发和比较了单模态和多模态模型，这些模型可以充分探索这两种类型的数据的预测能力。训练了一个 ResNet18 网络来对 3D CT 结节感兴趣区域（ROI）进行分类，而随机森林算法则用于对临床数据进行分类，前者的 AUC 为 0.7897，后者为 0.5241。对于多模态方法，实现了三种基于中间和晚期融合的策略，以组合 3D CT 结节 ROI 和临床数据的信息。在这些方法中，表现最佳的模型——一个接收临床数据和深度成像特征的组合作为输入的全连接层，该组合由 ResNet18 推断模型给出，其 AUC 为 0.8021。肺癌是一种复杂的疾病，其特征是存在多种生物学和生理学现象，并受到多种因素的影响。因此，模型必须能够应对这种需求。所获得的结果表明，不同类型的组合可能有潜力使模型对疾病进行更全面的分析。
