Department of Internal Medicine, UT Southwestern, Dallas, TX, USA.
Department of Population and Data Sciences, UT Southwestern, Dallas, TX, USA.
J Gen Intern Med. 2021 Sep;36(9):2555-2562. doi: 10.1007/s11606-020-06355-3. Epub 2021 Jan 14.
Electronic health record (EHR)-based readmission risk prediction models can be automated in real-time but have modest discrimination and may be missing important readmission risk factors. Clinician predictions of readmissions may incorporate information unavailable in the EHR, but the comparative usefulness is unknown. We sought to compare clinicians versus a validated EHR-based prediction model in predicting 30-day hospital readmissions.
We conducted a prospective survey of internal medicine clinicians in an urban safety-net hospital. Clinicians prospectively predicted patients' 30-day readmission risk on 5-point Likert scales, subsequently dichotomized into low- vs. high-risk. We compared human with machine predictions using discrimination, net reclassification, and diagnostic test characteristics. Observed readmissions were ascertained from a regional hospitalization database. We also developed and assessed a "human-plus-machine" logistic regression model incorporating both human and machine predictions.
We included 1183 hospitalizations from 106 clinicians, with a readmission rate of 20.8%. Both clinicians and the EHR model had similar discrimination (C-statistic 0.66 vs. 0.66, p = 0.91). Clinicians had higher specificity (79.0% vs. 48.9%, p < 0.001) but lower sensitivity (43.9 vs. 75.2%, p < 0.001) than EHR model predictions. Compared with machine, human was better at reclassifying non-readmissions (non-event NRI + 30.1%) but worse at reclassifying readmissions (event NRI - 31.3%). A human-plus-machine approach best optimized discrimination (C-statistic 0.70, 95% CI 0.67-0.74), sensitivity (65.5%), and specificity (66.7%).
Clinicians had similar discrimination but higher specificity and lower sensitivity than EHR model predictions. Human-plus-machine was better than either alone. Readmission risk prediction strategies should incorporate clinician assessments to optimize the accuracy of readmission predictions.
基于电子健康记录（EHR）的再入院风险预测模型可以实时自动化，但具有适度的区分能力，并且可能会遗漏重要的再入院风险因素。临床医生对再入院的预测可能会纳入 EHR 中无法获得的信息，但相对有用性尚不清楚。我们旨在比较临床医生与经过验证的基于 EHR 的预测模型在预测 30 天内医院再入院方面的表现。
我们对一家城市安全网医院的内科医生进行了前瞻性调查。临床医生前瞻性地使用 5 分李克特量表对患者的 30 天再入院风险进行预测，随后将其分为低风险和高风险。我们使用区分度、净再分类和诊断测试特征比较了人类与机器的预测结果。观察到的再入院情况是从区域住院数据库中确定的。我们还开发并评估了一种“人机结合”逻辑回归模型，该模型结合了人类和机器的预测。
我们纳入了来自 106 位临床医生的 1183 例住院患者，再入院率为 20.8%。临床医生和 EHR 模型的区分度相似（C 统计量分别为 0.66 和 0.66，p=0.91）。临床医生的特异性较高（79.0% vs. 48.9%，p<0.001），但敏感性较低（43.9% vs. 75.2%，p<0.001）。与 EHR 模型预测相比，人类更擅长重新分类非再入院患者（非事件 NRI+30.1%），但更不擅长重新分类再入院患者（事件 NRI-31.3%）。人机结合方法能够最佳优化区分度（C 统计量为 0.70，95%CI 为 0.67-0.74）、敏感性（65.5%）和特异性（66.7%）。
临床医生的区分度与 EHR 模型预测相似，但特异性更高，敏感性更低。人机结合方法优于单独使用任何一种方法。再入院风险预测策略应纳入临床医生评估，以优化再入院预测的准确性。
