School of Computer Science, Faculty of Engineering, The University of Sydney, NSW 2006, Australia.
School of Mathematics and Statistics, The University of Sydney, NSW 2006, Australia; Centre for Cancer Research, Westmead Institute of Medical Research, The University of Sydney, NSW 2145, Australia; Sydney Precision Data Science Centre, The University of Sydney, NSW 2006, Australia; Laboratory of Data Discovery for Health Limited (D(2)4H), Science Park, Hong Kong Special Administrative Region of China.
Comput Biol Med. 2023 Mar;154:106576. doi: 10.1016/j.compbiomed.2023.106576. Epub 2023 Feb 1.
The spatial architecture of the tumour microenvironment and phenotypic heterogeneity of tumour cells have been shown to be associated with cancer prognosis and clinical outcomes, including survival. Recent advances in highly multiplexed imaging, including imaging mass cytometry (IMC), capture spatially resolved, high-dimensional maps that quantify dozens of disease-relevant biomarkers at single-cell resolution, that contain potential to inform patient-specific prognosis. Existing automated methods for predicting survival, on the other hand, typically do not leverage spatial phenotype information captured at the single-cell level. Furthermore, there is no end-to-end method designed to leverage the rich information in whole IMC images and all marker channels, and aggregate this information with clinical data in a complementary manner to predict survival with enhanced accuracy. To that end, we present a deep multimodal graph-based network (DMGN) with two modules: (1) a multimodal graph-based module that considers relationships between spatial phenotype information in all image regions and all clinical variables adaptively, and (2) a clinical embedding module that automatically generates embeddings specialised for each clinical variable to enhance multimodal aggregation. We demonstrate that our modules are consistently effective at improving survival prediction performance using two public breast cancer datasets, and that our new approach can outperform state-of-the-art methods in survival prediction.
肿瘤微环境的空间结构和肿瘤细胞的表型异质性已被证明与癌症预后和临床结果（包括生存率）相关。高多重化成像技术的最新进展，包括成像质谱细胞术（IMC），可以捕获空间分辨率的高维图谱，以单细胞分辨率定量数十种与疾病相关的生物标志物，这些图谱具有提供患者特异性预后信息的潜力。另一方面，现有的用于预测生存率的自动化方法通常无法利用单细胞水平上捕获的空间表型信息。此外，没有端到端的方法可以利用整个 IMC 图像和所有标记通道中的丰富信息，并以互补的方式将这些信息与临床数据结合起来，以提高准确性来预测生存率。为此，我们提出了一个基于深度多模态图的网络（DMGN），它有两个模块：（1）一个基于多模态图的模块，它自适应地考虑所有图像区域和所有临床变量之间的空间表型信息关系；（2）一个临床嵌入模块，它自动为每个临床变量生成专门的嵌入，以增强多模态聚合。我们证明，我们的模块在使用两个公共乳腺癌数据集时，始终能够有效地提高生存率预测性能，并且我们的新方法在生存率预测方面可以优于最先进的方法。
