Respiratory Evaluation Sciences Program, Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, British Columbia, Canada.
Department of Epidemiology, CAPHRI Care and Public Health Research Institute, Maastricht University, Maastricht, The Netherlands.
Med Decis Making. 2023 Jul;43(5):564-575. doi: 10.1177/0272989X231178317. Epub 2023 Jun 22.
A previously developed risk prediction model needs to be validated before being used in a new population. The finite size of the validation sample entails that there is uncertainty around model performance. We apply value-of-information (VoI) methodology to quantify the consequence of uncertainty in terms of net benefit (NB).
We define the expected value of perfect information (EVPI) for model validation as the expected loss in NB due to not confidently knowing which of the alternative decisions confers the highest NB. We propose bootstrap-based and asymptotic methods for EVPI computations and conduct simulation studies to compare their performance. In a case study, we use the non-US subsets of a clinical trial as the development sample for predicting mortality after myocardial infarction and calculate the validation EVPI for the US subsample.
The computation methods generated similar EVPI values in simulation studies. EVPI generally declined with larger samples. In the case study, at the prespecified threshold of 0.02, the best decision with current information would be to use the model, with an incremental NB of 0.0020 over treating all. At this threshold, the EVPI was 0.0005 (relative EVPI = 25%). When scaled to the annual number of heart attacks in the US, the expected NB loss due to uncertainty was equal to 400 true positives or 19,600 false positives, indicating the value of further model validation.
VoI methods can be applied to the NB calculated during external validation of clinical prediction models. While uncertainty does not directly affect the clinical implications of NB findings, validation EVPI provides an objective perspective to the need for further validation and can be reported alongside NB in external validation studies.
External validation is a critical step when transporting a risk prediction model to a new setting, but the finite size of the validation sample creates uncertainty about the performance of the model.In decision theory, such uncertainty is associated with loss of net benefit because it can prevent one from identifying whether the use of the model is beneficial over alternative strategies.We define the expected value of perfect information for external validation as the expected loss in net benefit by not confidently knowing if the use of the model is net beneficial.The adoption of a model for a new population should be based on its expected net benefit; independently, value-of-information methods can be used to decide whether further validation studies are warranted.
在将先前开发的风险预测模型应用于新人群之前，需要对其进行验证。验证样本的有限大小意味着模型性能存在不确定性。我们应用信息价值（VoI）方法来量化不确定性对净收益（NB）的影响。
我们将模型验证的完美信息预期值（EVPI）定义为由于不确定模型使用是否带来最高 NB，从而导致 NB 损失的预期值。我们提出了基于 bootstrap 和渐近的 EVPI 计算方法，并进行了模拟研究以比较它们的性能。在一个案例研究中，我们将临床试验的非美国子集用作预测心肌梗死后死亡率的开发样本，并计算了美国子样本的验证 EVPI。
在模拟研究中，计算方法生成了相似的 EVPI 值。EVPI 通常随样本量的增大而减小。在案例研究中，在预设的 0.02 阈值下，当前信息的最佳决策是使用模型，相对于治疗所有患者，增量 NB 为 0.0020。在此阈值下，EVPI 为 0.0005（相对 EVPI=25%）。按美国每年心脏病发作的数量进行缩放，由于不确定性而导致的预期 NB 损失等于 400 个真阳性或 19600 个假阳性，表明需要进一步验证模型。
VoI 方法可应用于临床预测模型外部验证过程中计算的 NB。虽然不确定性不会直接影响 NB 结果的临床意义，但验证 EVPI 为进一步验证的必要性提供了客观视角，并可与外部验证研究中的 NB 一起报告。
将风险预测模型转移到新环境是一个关键步骤，但验证样本的有限大小会导致对模型性能的不确定性。在决策理论中，这种不确定性与净收益损失相关联，因为它可能阻止人们确定是否使用模型比替代策略更有益。我们将外部验证的完美信息预期值定义为由于不确定是否使用模型能带来净收益，从而导致的 NB 损失的预期值。对新人群采用模型应基于其预期的净收益；独立地，信息价值方法可用于决定是否进行进一步的验证研究。
