Gonzalez Ricardo, Nejat Peyman, Saha Ashirbani, Campbell Clinton J V, Norgan Andrew P, Lokker Cynthia
DeGroote School of Business, McMaster University, Hamilton, Ontario, Canada.
Division of Computational Pathology and Artificial Intelligence, Department of Laboratory Medicine and Pathology, Mayo Clinic, Rochester, MN, United States.
J Pathol Inform. 2023 Nov 5;15:100348. doi: 10.1016/j.jpi.2023.100348. eCollection 2024 Dec.
Numerous machine learning (ML) models have been developed for breast cancer using various types of data. Successful external validation (EV) of ML models is important evidence of their generalizability. The aim of this systematic review was to assess the performance of externally validated ML models based on histopathology images for diagnosis, classification, prognosis, or treatment outcome prediction in female breast cancer. A systematic search of MEDLINE, EMBASE, CINAHL, IEEE, MICCAI, and SPIE conferences was performed for studies published between January 2010 and February 2022. The Prediction Model Risk of Bias Assessment Tool (PROBAST) was employed, and the results were narratively described. Of the 2011 non-duplicated citations, 8 journal articles and 2 conference proceedings met inclusion criteria. Three studies externally validated ML models for diagnosis, 4 for classification, 2 for prognosis, and 1 for both classification and prognosis. Most studies used Convolutional Neural Networks and one used logistic regression algorithms. For diagnostic/classification models, the most common performance metrics reported in the EV were accuracy and area under the curve, which were greater than 87% and 90%, respectively, using pathologists' annotations/diagnoses as ground truth. The hazard ratios in the EV of prognostic ML models were between 1.7 (95% CI, 1.2-2.6) and 1.8 (95% CI, 1.3-2.7) to predict distant disease-free survival; 1.91 (95% CI, 1.11-3.29) for recurrence, and between 0.09 (95% CI, 0.01-0.70) and 0.65 (95% CI, 0.43-0.98) for overall survival, using clinical data as ground truth. Despite EV being an important step before the clinical application of a ML model, it hasn't been performed routinely. The large variability in the training/validation datasets, methods, performance metrics, and reported information limited the comparison of the models and the analysis of their results. Increasing the availability of validation datasets and implementing standardized methods and reporting protocols may facilitate future analyses.
已经使用各种类型的数据开发了许多用于乳腺癌的机器学习（ML）模型。ML模型的成功外部验证（EV）是其通用性的重要证据。本系统评价的目的是评估基于组织病理学图像的外部验证ML模型在女性乳腺癌诊断、分类、预后或治疗结果预测方面的性能。对MEDLINE、EMBASE、CINAHL、IEEE、MICCAI和SPIE会议进行了系统检索，以查找2010年1月至2022年2月发表的研究。采用预测模型偏倚风险评估工具（PROBAST），并对结果进行叙述性描述。在2011条非重复引用中，8篇期刊文章和2篇会议论文符合纳入标准。三项研究对诊断的ML模型进行了外部验证，四项对分类进行了验证，两项对预后进行了验证，一项对分类和预后均进行了验证。大多数研究使用卷积神经网络，一项使用逻辑回归算法。对于诊断/分类模型，在外部验证中报告的最常见性能指标是准确率和曲线下面积，以病理学家的注释/诊断作为金标准时，分别大于87%和90%。预后ML模型在外部验证中的风险比在预测远处无病生存时为1.7（95%CI，1.2 - 2.6）至1.8（95%CI，1.3 - 2.7）；复发时为1.91（95%CI，1.11 - 3.29），总生存时为0.09（95%CI，0.01 - 0.70）至0.65（95%CI，0.43 - 0.98），以临床数据作为金标准。尽管外部验证是ML模型临床应用前的重要一步，但尚未常规进行。训练/验证数据集、方法、性能指标和报告信息的巨大差异限制了模型的比较及其结果的分析。增加验证数据集的可用性并实施标准化方法和报告协议可能有助于未来的分析。
