Grolleau François, Tibshirani Robert, Chen Jonathan H
Stanford Center for Biomedical Informatics Research, Stanford University.
Department of Statistics, Stanford University.
AMIA Jt Summits Transl Sci Proc. 2025 Jun 10;2025:196-204. eCollection 2025.
Rigorous external validation is crucial for assessing the generalizability of prediction models, particularly by evaluating their discrimination (AUROC) on new data. This often involves comparing a new model's AUROC to that of an established reference model. However, many studies rely on arbitrary rules of thumb for sample size calculations, often resulting in underpowered analyses and unreliable conclusions. This paper reviews crucial concepts for accurate sample size determination in AUROC-based external validation studies, making the theory and practice more accessible to researchers and clinicians. We introduce powerROC, an open-source web tool designed to simplify these calculations, enabling both the evaluation of a single model and the comparison of two models. The tool offers guidance on selecting target precision levels and employs flexible approaches, leveraging either pilot data or user-defined probability distributions. We illustrate powerROC's utility through a case study on hospital mortality prediction using the MIMIC database.
严格的外部验证对于评估预测模型的通用性至关重要，特别是通过在新数据上评估其区分度（AUROC）。这通常涉及将新模型的AUROC与既定参考模型的AUROC进行比较。然而，许多研究在样本量计算方面依赖任意的经验法则，常常导致分析效能不足和结论不可靠。本文回顾了基于AUROC的外部验证研究中准确确定样本量的关键概念，使研究人员和临床医生更容易理解相关理论和实践。我们介绍了powerROC，这是一个开源网络工具，旨在简化这些计算，能够对单个模型进行评估并比较两个模型。该工具为选择目标精度水平提供指导，并采用灵活的方法，利用试点数据或用户定义的概率分布。我们通过使用MIMIC数据库进行医院死亡率预测的案例研究来说明powerROC的效用。
