Department of Orthopaedic Surgery, National Taiwan University Hospital, Taipei, Taiwan.
Graduate Institute of Biomedical Electronics and Bioinformatics, National Taiwan University, Taipei, Taiwan.
Clin Orthop Relat Res. 2024 Dec 1;482(12):2193-2208. doi: 10.1097/CORR.0000000000003185. Epub 2024 Jul 23.
Survival estimation for patients with symptomatic skeletal metastases ideally should be made before a type of local treatment has already been determined. Currently available survival prediction tools, however, were generated using data from patients treated either operatively or with local radiation alone, raising concerns about whether they would generalize well to all patients presenting for assessment. The Skeletal Oncology Research Group machine-learning algorithm (SORG-MLA), trained with institution-based data of surgically treated patients, and the Metastases location, Elderly, Tumor primary, Sex, Sickness/comorbidity, and Site of radiotherapy model (METSSS), trained with registry-based data of patients treated with radiotherapy alone, are two of the most recently developed survival prediction models, but they have not been tested on patients whose local treatment strategy is not yet decided.
QUESTIONS/PURPOSES: (1) Which of these two survival prediction models performed better in a mixed cohort made up both of patients who received local treatment with surgery followed by radiotherapy and who had radiation alone for symptomatic bone metastases? (2) Which model performed better among patients whose local treatment consisted of only palliative radiotherapy? (3) Are laboratory values used by SORG-MLA, which are not included in METSSS, independently associated with survival after controlling for predictions made by METSSS?
Between 2010 and 2018, we provided local treatment for 2113 adult patients with skeletal metastases in the extremities at an urban tertiary referral academic medical center using one of two strategies: (1) surgery followed by postoperative radiotherapy or (2) palliative radiotherapy alone. Every patient's survivorship status was ascertained either by their medical records or the national death registry from the Taiwanese National Health Insurance Administration. After applying a priori designated exclusion criteria, 91% (1920) were analyzed here. Among them, 48% (920) of the patients were female, and the median (IQR) age was 62 years (53 to 70 years). Lung was the most common primary tumor site (41% [782]), and 59% (1128) of patients had other skeletal metastases in addition to the treated lesion(s). In general, the indications for surgery were the presence of a complete pathologic fracture or an impending pathologic fracture, defined as having a Mirels score of ≥ 9, in patients with an American Society of Anesthesiologists (ASA) classification of less than or equal to IV and who were considered fit for surgery. The indications for radiotherapy were relief of pain, local tumor control, prevention of skeletal-related events, and any combination of the above. In all, 84% (1610) of the patients received palliative radiotherapy alone as local treatment for the target lesion(s), and 16% (310) underwent surgery followed by postoperative radiotherapy. Neither METSSS nor SORG-MLA was used at the point of care to aid clinical decision-making during the treatment period. Survival was retrospectively estimated by these two models to test their potential for providing survival probabilities. We first compared SORG to METSSS in the entire population. Then, we repeated the comparison in patients who received local treatment with palliative radiation alone. We assessed model performance by area under the receiver operating characteristic curve (AUROC), calibration analysis, Brier score, and decision curve analysis (DCA). The AUROC measures discrimination, which is the ability to distinguish patients with the event of interest (such as death at a particular time point) from those without. AUROC typically ranges from 0.5 to 1.0, with 0.5 indicating random guessing and 1.0 a perfect prediction, and in general, an AUROC of ≥ 0.7 indicates adequate discrimination for clinical use. Calibration refers to the agreement between the predicted outcomes (in this case, survival probabilities) and the actual outcomes, with a perfect calibration curve having an intercept of 0 and a slope of 1. A positive intercept indicates that the actual survival is generally underestimated by the prediction model, and a negative intercept suggests the opposite (overestimation). When comparing models, an intercept closer to 0 typically indicates better calibration. Calibration can also be summarized as log(O:E), the logarithm scale of the ratio of observed (O) to expected (E) survivors. A log(O:E) > 0 signals an underestimation (the observed survival is greater than the predicted survival); and a log(O:E) < 0 indicates the opposite (the observed survival is lower than the predicted survival). A model with a log(O:E) closer to 0 is generally considered better calibrated. The Brier score is the mean squared difference between the model predictions and the observed outcomes, and it ranges from 0 (best prediction) to 1 (worst prediction). The Brier score captures both discrimination and calibration, and it is considered a measure of overall model performance. In Brier score analysis, the "null model" assigns a predicted probability equal to the prevalence of the outcome and represents a model that adds no new information. A prediction model should achieve a Brier score at least lower than the null-model Brier score to be considered as useful. The DCA was developed as a method to determine whether using a model to inform treatment decisions would do more good than harm. It plots the net benefit of making decisions based on the model's predictions across all possible risk thresholds (or cost-to-benefit ratios) in relation to the two default strategies of treating all or no patients. The care provider can decide on an acceptable risk threshold for the proposed treatment in an individual and assess the corresponding net benefit to determine whether consulting with the model is superior to adopting the default strategies. Finally, we examined whether laboratory data, which were not included in the METSSS model, would have been independently associated with survival after controlling for the METSSS model's predictions by using the multivariable logistic and Cox proportional hazards regression analyses.
Between the two models, only SORG-MLA achieved adequate discrimination (an AUROC of > 0.7) in the entire cohort (of patients treated operatively or with radiation alone) and in the subgroup of patients treated with palliative radiotherapy alone. SORG-MLA outperformed METSSS by a wide margin on discrimination, calibration, and Brier score analyses in not only the entire cohort but also the subgroup of patients whose local treatment consisted of radiotherapy alone. In both the entire cohort and the subgroup, DCA demonstrated that SORG-MLA provided more net benefit compared with the two default strategies (of treating all or no patients) and compared with METSSS when risk thresholds ranged from 0.2 to 0.9 at both 90 days and 1 year, indicating that using SORG-MLA as a decision-making aid was beneficial when a patient's individualized risk threshold for opting for treatment was 0.2 to 0.9. Higher albumin, lower alkaline phosphatase, lower calcium, higher hemoglobin, lower international normalized ratio, higher lymphocytes, lower neutrophils, lower neutrophil-to-lymphocyte ratio, lower platelet-to-lymphocyte ratio, higher sodium, and lower white blood cells were independently associated with better 1-year and overall survival after adjusting for the predictions made by METSSS.
Based on these discoveries, clinicians might choose to consult SORG-MLA instead of METSSS for survival estimation in patients with long-bone metastases presenting for evaluation of local treatment. Basing a treatment decision on the predictions of SORG-MLA could be beneficial when a patient's individualized risk threshold for opting to undergo a particular treatment strategy ranged from 0.2 to 0.9. Future studies might investigate relevant laboratory items when constructing or refining a survival estimation model because these data demonstrated prognostic value independent of the predictions of the METSSS model, and future studies might also seek to keep these models up to date using data from diverse, contemporary patients undergoing both modern operative and nonoperative treatments.
Level III, diagnostic study.
对于有症状的骨骼转移患者，在确定某种局部治疗方案之前，最好对其生存估计进行预测。然而，目前可用的生存预测工具是基于接受手术或单纯放疗的患者数据生成的，这引发了人们的担忧，即这些工具是否能够很好地推广到所有接受评估的患者。骨骼肿瘤研究组机器学习算法（SORG-MLA）和转移部位、年龄、肿瘤原发部位、性别、疾病严重程度/合并症、放疗部位模型（METSSS）是最近开发的两种生存预测模型，但它们尚未在局部治疗方案尚未确定的患者中进行测试。
问题/目的：（1）在由接受手术联合放疗和单纯放疗的患者组成的混合队列中，这两种生存预测模型中哪一种的表现更好？（2）在局部治疗仅为单纯放疗的患者中，哪种模型的表现更好？（3）SORG-MLA 中使用的实验室值是否与 METSSS 预测无关，并与 METSSS 预测后患者的生存相关？
2010 年至 2018 年期间，我们在一家城市三级转诊学术医疗中心对 2113 例四肢骨骼转移的成年患者提供了局部治疗，采用以下两种策略之一：（1）手术联合术后放疗或（2）单纯放疗。每位患者的生存状态通过病历或台湾全民健康保险署的全国死亡登记来确定。在应用预先指定的排除标准后，91%（1920 例）患者被纳入分析。其中，48%（920 例）为女性，中位（IQR）年龄为 62 岁（53 岁至 70 岁）。肺是最常见的原发肿瘤部位（41%[782 例]），除了治疗部位外，还有 59%（1128 例）的患者有其他骨骼转移。一般来说，手术的适应证是存在完全病理性骨折或即将发生的病理性骨折，定义为 Mirels 评分≥9，美国麻醉师协会（ASA）分级≤Ⅳ级，且患者适合手术。放疗的适应证是缓解疼痛、局部肿瘤控制、预防骨骼相关事件，以及上述任何一种的联合。共有 84%（1610 例）的患者接受单纯放疗作为目标病变的局部治疗，16%（310 例）接受手术联合术后放疗。在治疗期间，METSSS 和 SORG-MLA 都未用于指导临床决策。这两种模型被用来回顾性地估计生存情况，以检验它们提供生存概率的潜力。我们首先比较了 SORG 与 METSSS 在整个队列中的表现。然后，我们在接受单纯放疗的患者中重复了这一比较。我们通过受试者工作特征曲线（ROC）下面积（AUROC）、校准分析、Brier 评分和决策曲线分析（DCA）来评估模型性能。AUROC 衡量的是区分能力，即区分有特定时间点死亡事件的患者与无该事件的患者的能力。AUROC 通常在 0.5 到 1.0 之间，0.5 表示随机猜测，1.0 表示完美预测，一般来说，AUROC≥0.7 表示临床使用的区分能力足够。校准是指预测结果（在这种情况下是生存概率）与实际结果之间的一致性，完美的校准曲线截距为 0，斜率为 1。正截距表示实际生存概率普遍被预测模型低估，负截距则相反（高估）。在比较模型时，截距越接近 0，通常表示校准越好。校准也可以总结为对数（O:E），即观察（O）与预期（E）幸存者的比例的对数。对数（O:E）>0 表示低估（观察到的生存时间大于预测的生存时间）；对数（O:E）<0 表示相反（观察到的生存时间低于预测的生存时间）。对数（O:E）更接近 0 的模型通常被认为校准更好。Brier 评分是模型预测值与观察到的结果之间的均方差差异，范围从 0（最佳预测）到 1（最差预测）。Brier 评分同时捕捉了区分度和校准度，它是衡量模型整体性能的一个指标。在 Brier 评分分析中，“零模型”将预测概率分配给结果的患病率，代表一种不增加任何新信息的模型。预测模型的 Brier 评分至少要低于零模型的 Brier 评分，才能被认为是有用的。决策曲线分析（DCA）是一种方法，用于确定根据模型的预测来做出治疗决策是否会带来更多的好处而不是坏处。它在所有可能的风险阈值（或成本效益比）范围内绘制了基于模型预测的决策的净收益，以及治疗所有患者或不治疗任何患者的两种默认策略之间的关系。临床医生可以根据患者的个体情况确定可接受的治疗风险阈值，并评估相应的净收益，以确定是否咨询模型优于采用默认策略。最后，我们通过多变量逻辑回归和 Cox 比例风险回归分析，检验了 METSSS 模型中未包含的实验室数据是否与 METSSS 模型的预测无关，并且与生存相关。
在两种模型中，只有 SORG-MLA 在接受手术或单纯放疗的患者的整个队列和单纯放疗的患者亚组中均达到了足够的区分度（AUROC>0.7）。在整个队列和单纯放疗的患者亚组中，SORG-MLA 在区分度、校准度和 Brier 评分分析方面均优于 METSSS，表明在风险阈值范围为 0.2 至 0.9 时，与两种默认策略（治疗所有或不治疗任何患者）相比，SORG-MLA 作为决策辅助工具具有优势，与 METSSS 相比，当患者选择治疗的个体化风险阈值为 0.2 至 0.9 时，SORG-MLA 作为决策辅助工具具有优势。较高的白蛋白、较低的碱性磷酸酶、较低的钙、较高的血红蛋白、较低的国际标准化比值、较高的淋巴细胞、较低的中性粒细胞、较低的中性粒细胞/淋巴细胞比值、较低的血小板/淋巴细胞比值、较高的钠和较低的白细胞与 1 年和总体生存率的提高独立相关，校正 METSSS 预测后。
基于这些发现，临床医生可能会选择咨询 SORG-MLA 而不是 METSSS 来估计有长骨转移的患者接受局部治疗的生存情况。当患者选择特定治疗策略的个体化风险阈值在 0.2 到 0.9 之间时，基于 SORG-MLA 预测做出的治疗决策可能会带来益处。未来的研究可能会研究相关的实验室项目，因为这些数据独立于 METSSS 模型的预测显示出预后价值，并可能进一步寻求使用来自接受现代手术和非手术治疗的不同当代患者的数据来更新和改进生存估计模型。
III 级，诊断性研究。
