Janssen Research and Development, Titusville, NJ, USA; Observational Health Data Sciences and Informatics (OHDSI), New York, NY.
Janssen Research and Development, Titusville, NJ, USA; Observational Health Data Sciences and Informatics (OHDSI), New York, NY.
J Biomed Inform. 2022 Nov;135:104177. doi: 10.1016/j.jbi.2022.104177. Epub 2022 Aug 19.
Phenotype algorithms are central to performing analyses using observational data. These algorithms translate the clinical idea of a health condition into an executable set of rules allowing for queries of data elements from a database. PheValuator, a software package in the Observational Health Data Sciences and Informatics (OHDSI) tool stack, provides a method to assess the performance characteristics of these algorithms, namely, sensitivity, specificity, and positive and negative predictive value. It uses machine learning to develop predictive models for determining a probabilistic gold standard of subjects for assessment of cases and non-cases of health conditions. PheValuator was developed to complement or even replace the traditional approach of algorithm validation, i.e., by expert assessment of subject records through chart review. Results in our first PheValuator paper suggest a systematic underestimation of the PPV compared to previous results using chart review. In this paper we evaluate modifications made to the method designed to improve its performance.
The major changes to PheValuator included allowing all diagnostic conditions, clinical observations, drug prescriptions, and laboratory measurements to be included as predictors within the modeling process whereas in the prior version there were significant restrictions on the included predictors. We also have allowed for the inclusion of the temporal relationships of the predictors in the model. To evaluate the performance of the new method, we compared the results from the new and original methods against results found from the literature using traditional validation of algorithms for 19 phenotypes. We performed these tests using data from five commercial databases.
In the assessment aggregating all phenotype algorithms, the median difference between the PheValuator estimate and the gold standard estimate for PPV was reduced from -21 (IQR -34, -3) in Version 1.0 to 4 (IQR -3, 15) using Version 2.0. We found a median difference in specificity of 3 (IQR 1, 4.25) for Version 1.0 and 3 (IQR 1, 4) for Version 2.0. The median difference between the two versions of PheValuator and the gold standard for estimates of sensitivity was reduced from -39 (-51, -20) to -16 (-34, -6).
PheValuator 2.0 produces estimates for the performance characteristics for phenotype algorithms that are significantly closer to estimates from traditional validation through chart review compared to version 1.0. With this tool in researcher's toolkits, methods, such as quantitative bias analysis, may now be used to improve the reliability and reproducibility of research studies using observational data.
表型算法是使用观察数据进行分析的核心。这些算法将健康状况的临床概念转化为可执行的规则集，允许对数据库中的数据元素进行查询。在观察性健康数据科学和信息学（OHDSI）工具集中，PheValuator 软件包提供了一种评估这些算法性能特征的方法，即敏感性、特异性、阳性和阴性预测值。它使用机器学习为确定健康状况病例和非病例的受试者概率金标准开发预测模型。PheValuator 的开发旨在补充甚至替代传统的算法验证方法，即通过专家对病历进行图表审查来评估。我们的第一篇 PheValuator 论文中的结果表明，与使用图表审查的先前结果相比，PPV 存在系统性低估。在本文中，我们评估了为提高其性能而对方法进行的修改。
对 PheValuator 的主要更改包括允许将所有诊断条件、临床观察、药物处方和实验室测量作为预测因子包含在建模过程中，而在以前的版本中，对包含的预测因子有重大限制。我们还允许在模型中包含预测因子的时间关系。为了评估新方法的性能，我们将新方法和原始方法的结果与使用传统算法验证的 19 种表型的文献结果进行了比较。我们使用来自五个商业数据库的数据进行了这些测试。
在评估汇总所有表型算法的结果时，使用新版本 2.0 时，PPV 的 PheValuator 估计值与黄金标准估计值之间的中位数差异从版本 1.0 的-21（IQR-34，-3）减少到 4（IQR-3，15）。我们发现版本 1.0 的特异性中位数差异为 3（IQR1,4.25），版本 2.0 的特异性中位数差异为 3（IQR1,4）。与旧版本相比，PheValuator 新版本 2.0 与黄金标准之间的敏感性估计值的中位数差异从-39（-51，-20）减少到-16（-34，-6）。
与版本 1.0 相比，PheValuator 2.0 生成的表型算法性能特征估计值与通过图表审查进行的传统验证估计值更为接近。有了这个工具在研究人员的工具包中，现在可以使用定量偏差分析等方法来提高使用观察数据的研究的可靠性和可重复性。
