Department of Medical Physics, Guy's and St Thomas' NHS Foundation Trust, London, United Kingdom.
School of Cancer and Pharmaceutical Sciences, Comprehensive Cancer Centre, King's College London, London, United Kingdom.
Phys Med Biol. 2024 Oct 10;69(20). doi: 10.1088/1361-6560/ad8290.
Normal tissue complication probability (NTCP) modelling is rapidly embracing deep learning (DL) methods, acknowledging the importance of spatial dose information. Finding effective ways to combine information from radiation dose distribution maps (dosiomics) and clinical data involves technical challenges and requires domain knowledge. We propose different multi-modality data fusion strategies to facilitate future DL-based NTCP studies.Early, joint and late DL multi-modality fusion strategies were compared using clinical and mandibular radiation dose distribution volumes. These were contrasted with single-modality models: a random forest trained on non-image data (clinical, demographic and dose-volume metrics) and a 3D DenseNet-40 trained on image data (mandibular dose distribution maps). The study involved a matched cohort of 92 osteoradionecrosis cases and 92 controls from a single institution.The late fusion model exhibited superior discrimination and calibration performance, while the join fusion achieved a more balanced distribution of the predicted probabilities. Discrimination performance did not significantly differ between strategies. Late fusion, though less technically complex, lacks crucial inter-modality interactions for NTCP modelling. In contrast, joint fusion, despite its complexity, resulted in a single network training process which included intra- and inter-modality interactions in its model parameter optimisation.This study is a pioneering effort in comparing different strategies for including image data into DL-based NTCP models in combination with lower dimensional data such as clinical variables. The discrimination performance of such multi-modality NTCP models and the choice of fusion strategy will depend on the distribution and quality of both types of data. Multiple data fusion strategies should be compared and reported in multi-modality NTCP modelling using DL.
正常组织并发症概率 (NTCP) 建模正在迅速采用深度学习 (DL) 方法，承认空间剂量信息的重要性。寻找有效结合辐射剂量分布图 (剂量组学) 和临床数据信息的方法涉及技术挑战，并需要领域知识。我们提出了不同的多模态数据融合策略，以促进未来基于 DL 的 NTCP 研究。使用临床和下颌辐射剂量分布体积比较了早期、联合和晚期 DL 多模态融合策略。这些策略与单模态模型进行了对比：一个基于非图像数据（临床、人口统计学和剂量-体积指标）的随机森林和一个基于图像数据（下颌剂量分布图）的 3D DenseNet-40。该研究涉及来自单一机构的 92 例放射性骨坏死病例和 92 例对照的匹配队列。晚期融合模型表现出卓越的区分和校准性能，而联合融合则实现了预测概率的更平衡分布。策略之间的区分性能没有显著差异。尽管晚期融合技术上不太复杂，但缺乏 NTCP 建模所需的关键模态间相互作用。相比之下，尽管联合融合很复杂，但它导致了一个单一的网络训练过程，在其模型参数优化中包括了模态内和模态间的相互作用。这项研究是在比较不同策略将图像数据纳入基于 DL 的 NTCP 模型与临床变量等低维数据相结合方面的开创性努力。这种多模态 NTCP 模型的区分性能和融合策略的选择将取决于这两种类型数据的分布和质量。在使用 DL 进行多模态 NTCP 建模时，应比较和报告多种数据融合策略。
