Janssen Research and Development, 1125 Trenton Harbourton Rd, Titusville, NJ, 08560, USA.
Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, The Netherlands.
BMC Med Res Methodol. 2020 May 6;20(1):102. doi: 10.1186/s12874-020-00991-3.
To demonstrate how the Observational Healthcare Data Science and Informatics (OHDSI) collaborative network and standardization can be utilized to scale-up external validation of patient-level prediction models by enabling validation across a large number of heterogeneous observational healthcare datasets.
Five previously published prognostic models (ATRIA, CHADS, CHADSVASC, Q-Stroke and Framingham) that predict future risk of stroke in patients with atrial fibrillation were replicated using the OHDSI frameworks. A network study was run that enabled the five models to be externally validated across nine observational healthcare datasets spanning three countries and five independent sites.
The five existing models were able to be integrated into the OHDSI framework for patient-level prediction and they obtained mean c-statistics ranging between 0.57-0.63 across the 6 databases with sufficient data to predict stroke within 1 year of initial atrial fibrillation diagnosis for females with atrial fibrillation. This was comparable with existing validation studies. The validation network study was run across nine datasets within 60 days once the models were replicated. An R package for the study was published at https://github.com/OHDSI/StudyProtocolSandbox/tree/master/ExistingStrokeRiskExternalValidation.
This study demonstrates the ability to scale up external validation of patient-level prediction models using a collaboration of researchers and a data standardization that enable models to be readily shared across data sites. External validation is necessary to understand the transportability or reproducibility of a prediction model, but without collaborative approaches it can take three or more years for a model to be validated by one independent researcher. In this paper we show it is possible to both scale-up and speed-up external validation by showing how validation can be done across multiple databases in less than 2 months. We recommend that researchers developing new prediction models use the OHDSI network to externally validate their models.
为了展示 Observational Healthcare Data Science and Informatics（OHDSI）协作网络和标准化如何被利用来扩大患者水平预测模型的外部验证规模，使验证能够跨越大量异构的观察性医疗保健数据集。
使用 OHDSI 框架复制了五个先前发表的预测模型（ATRIA、CHADS、CHADSVASC、Q-Stroke 和 Framingham），这些模型预测心房颤动患者未来中风风险。进行了一项网络研究，使五个模型能够在跨越三个国家和五个独立站点的九个观察性医疗保健数据集上进行外部验证。
五个现有模型能够集成到 OHDSI 框架中进行患者水平预测，并且在六个数据库中，它们在预测女性心房颤动患者 1 年内初始心房颤动诊断后中风风险方面获得了 0.57-0.63 的平均 c 统计量。这与现有的验证研究相当。一旦模型被复制，在 60 天内就可以在九个数据集上运行验证网络研究。该研究的 R 包已在 https://github.com/OHDSI/StudyProtocolSandbox/tree/master/ExistingStrokeRiskExternalValidation 上发布。
本研究展示了使用研究人员合作和数据标准化来扩大患者水平预测模型的外部验证能力，使模型能够在数据站点之间轻松共享。外部验证对于理解预测模型的可转移性或可再现性是必要的，但如果没有合作方法，一个独立研究人员可能需要三年或更长时间才能验证一个模型。在本文中，我们通过展示如何在不到 2 个月的时间内在多个数据库中进行验证，展示了如何扩大和加速外部验证，证明了这一点。我们建议开发新预测模型的研究人员使用 OHDSI 网络来对其模型进行外部验证。
