Wang L E, Shaw Pamela A, Mathelier Hansie M, Kimmel Stephen E, French Benjamin
DEPARTMENT OF BIOSTATISTICS AND EPIDEMIOLOGY, UNIVERSITY OF PENNSYLVANIA, 423 GUARDIAN DRIVE, PHILADELPHIA, PENNSYLVANIA 19104, USA.
DEPARTMENT OF MEDICINE, UNIVERSITY OF PENNSYLVANIA, 51 N 39TH STREET, PHILADELPHIA, PENNSYLVANIA 19104, USA.
Ann Appl Stat. 2016 Mar;10(1):286-304. doi: 10.1214/15-AOAS891.
The availability of data from electronic health records facilitates the development and evaluation of risk-prediction models, but estimation of prediction accuracy could be limited by outcome misclassification, which can arise if events are not captured. We evaluate the robustness of prediction accuracy summaries, obtained from receiver operating characteristic curves and risk-reclassification methods, if events are not captured (i.e., "false negatives"). We derive estimators for sensitivity and specificity if misclassification is independent of marker values. In simulation studies, we quantify the potential for bias in prediction accuracy summaries if misclassification depends on marker values. We compare the accuracy of alternative prognostic models for 30-day all-cause hospital readmission among 4548 patients discharged from the University of Pennsylvania Health System with a primary diagnosis of heart failure. Simulation studies indicate that if misclassification depends on marker values, then the estimated accuracy improvement is also biased, but the direction of the bias depends on the direction of the association between markers and the probability of misclassification. In our application, 29% of the 1143 readmitted patients were readmitted to a hospital elsewhere in Pennsylvania, which reduced prediction accuracy. Outcome misclassification can result in erroneous conclusions regarding the accuracy of risk-prediction models.
电子健康记录中的数据可用性促进了风险预测模型的开发和评估，但预测准确性的估计可能会受到结局误分类的限制，如果事件未被捕获，就可能出现这种情况。我们评估了从接受者操作特征曲线和风险重新分类方法获得的预测准确性总结在事件未被捕获（即“假阴性”）时的稳健性。如果误分类与标志物值无关，我们推导出敏感性和特异性的估计量。在模拟研究中，如果误分类取决于标志物值，我们量化预测准确性总结中偏差的可能性。我们比较了宾夕法尼亚大学医疗系统出院的4548例原发性心力衰竭患者30天全因再入院的替代预后模型的准确性。模拟研究表明，如果误分类取决于标志物值，那么估计的准确性改善也会有偏差，但偏差的方向取决于标志物与误分类概率之间关联的方向。在我们的应用中，1143例再入院患者中有29%在宾夕法尼亚州的其他医院再入院，这降低了预测准确性。结局误分类可能导致关于风险预测模型准确性的错误结论。
