Song Bolin, Leroy Amaury, Yang Kailin, Dam Tanmoy, Wang Xiangxue, Maurya Himanshu, Pathak Tilak, Lee Jonathan, Stock Sarah, Li Xiao T, Fu Pingfu, Lu Cheng, Toro Paula, Chute Deborah J, Koyfman Shlomo, Saba Nabil F, Patel Mihir R, Madabhushi Anant
Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology and Emory University, Atlanta, GA, USA.
Therapanacea, Paris, France.
EBioMedicine. 2025 Apr;114:105663. doi: 10.1016/j.ebiom.2025.105663. Epub 2025 Mar 22.
We aim to predict outcomes of human papillomavirus (HPV)-associated oropharyngeal squamous cell carcinoma (OPSCC), a subtype of head and neck cancer characterized with improved clinical outcome and better response to therapy. Pathology and radiology focused AI-based prognostic models have been independently developed for OPSCC, but their integration incorporating both primary tumour (PT) and metastatic cervical lymph node (LN) remains unexamined.
We investigate the prognostic value of an AI approach termed the swintransformer-based multimodal and multi-region data fusion framework (SMuRF). SMuRF integrates features from CT corresponding to the PT and LN, as well as whole slide pathology images from the PT as a predictor of survival and tumour grade in HPV-associated OPSCC. SMuRF employs cross-modality and cross-region window based multi-head self-attention mechanisms to capture interactions between features across tumour habitats and image scales.
Developed and tested on a cohort of 277 patients with OPSCC with matched radiology and pathology images, SMuRF demonstrated strong performance (C-index = 0.81 for DFS prediction and AUC = 0.75 for tumour grade classification) and emerged as an independent prognostic biomarker for DFS (hazard ratio [HR] = 17, 95% confidence interval [CI], 4.9-58, p < 0.0001) and tumour grade (odds ratio [OR] = 3.7, 95% CI, 1.4-10.5, p = 0.01) controlling for other clinical variables (i.e., T-, N-stage, age, smoking, sex and treatment modalities). Importantly, SMuRF outperformed unimodal models derived from radiology or pathology alone.
Our findings underscore the potential of multimodal deep learning in accurately stratifying OPSCC risk, informing tailored treatment strategies and potentially refining existing treatment algorithms.
The National Institutes of Health, the U.S. Department of Veterans Affairs and National Institute of Biomedical Imaging and Bioengineering.
我们旨在预测人乳头瘤病毒（HPV）相关的口咽鳞状细胞癌（OPSCC）的预后，这是一种头颈癌亚型，其临床预后有所改善，对治疗的反应也更好。针对OPSCC，已经分别开发了基于病理学和放射学的人工智能预后模型，但它们对原发性肿瘤（PT）和转移性颈部淋巴结（LN）的整合情况尚未得到研究。
我们研究了一种名为基于swin变压器的多模态多区域数据融合框架（SMuRF）的人工智能方法的预后价值。SMuRF整合了来自PT和LN的CT特征，以及来自PT的全切片病理图像，作为HPV相关OPSCC生存和肿瘤分级的预测指标。SMuRF采用基于跨模态和跨区域窗口的多头自注意力机制，以捕捉肿瘤部位和图像尺度之间特征的相互作用。
在一组277例具有匹配放射学和病理学图像的OPSCC患者中进行开发和测试，SMuRF表现出强大的性能（DFS预测的C指数 = 0.81，肿瘤分级分类的AUC = 0.75），并成为DFS（风险比[HR] = 17，95%置信区间[CI]，4.9 - 58，p < 0.0001）和肿瘤分级（优势比[OR] = 3.7，95% CI，1.4 - 10.5，p = 0.01）的独立预后生物标志物，同时控制了其他临床变量（即T分期、N分期、年龄、吸烟、性别和治疗方式）。重要的是，SMuRF优于仅从放射学或病理学得出的单模态模型。
我们的研究结果强调了多模态深度学习在准确分层OPSCC风险、为量身定制的治疗策略提供信息以及潜在优化现有治疗算法方面的潜力。
美国国立卫生研究院、美国退伍军人事务部和国家生物医学成像和生物工程研究所。
