Department of Internal Medicine, Division of General Internal Medicine and Geriatrics, Oregon Health & Science University, School of Medicine, 3181 SW Sam Jackson Park Rd. Mail Code: L-475, Portland, OR, 97239, USA.
Department of Clinical Informatics and Clinical Epidemiology, Oregon Health & Science University, School of Medicine, Portland, USA.
J Gen Intern Med. 2022 Feb;37(3):601-607. doi: 10.1007/s11606-021-06896-1. Epub 2021 Jun 7.
In primary care risk stratification, automated algorithms do not consider the same factors as providers. The process of adjudication, in which providers review and adjust algorithm-derived risk scores, may improve the prediction of adverse outcomes.
We assessed the patient factors that influenced provider adjudication behavior and evaluated the performance of an adjudicated risk model against a commercial algorithm.
(1) Structured interviews with primary care providers (PCP) and multivariable regression analysis and (2) receiver operating characteristic curves (ROC) with sensitivity analyses.
Primary care patients aged 18 years and older with an adjudicated risk score. APPROACH AND MAIN MEASURES: (1) Themes from structured interviews and discrete variables associated with provider adjudication behavior; (2) comparison of concordance statistics and sensitivities between risk models.
47,940 patients were adjudicated by PCPs in 2018. Interviews revealed that, in adjudication, providers consider disease severity, presence of self-management skills, behavioral health, and whether a risk score is actionable. Provider up-scoring from the algorithmic risk score was significantly associated with patient male sex (OR 1.24, CI 1.15-1.34), age > 65 (OR 2.55, CI 2.24-2.91), Black race (1.26, CI 1.02-1.55), polypharmacy >10 medications (OR 4.87, CI 4.27-5.56), a positive depression screen (OR 1.57, CI 1.43-1.72), and hemoglobin A1c >9 (OR 1.89, CI 1.52-2.33). Overall, the adjudicated risk model performed better than the commercial algorithm for all outcomes: ED visits (c-statistic 0.689 vs. 0.684, p < 0.01), hospital admissions (c-statistic 0.663 vs. 0.649, p < 0.01), and death (c-statistic 0.753 vs. 0.721, p < 0.01). When limited to males or seniors, the adjudicated models displayed either improved or non-inferior performance compared to the commercial model.
Provider adjudication of risk stratification improves model performance because providers have a personal understanding of their patients and are able to apply their training to clinical decision-making.
在初级保健风险分层中，自动化算法并未考虑到与提供者相同的因素。通过对算法推导的风险评分进行审查和调整的裁决过程，可能会提高不良结果的预测能力。
我们评估了影响提供者裁决行为的患者因素，并评估了经裁决的风险模型与商业算法的性能。
（1）对初级保健提供者进行结构访谈和多变量回归分析；（2）接收者操作特征曲线（ROC）和敏感性分析。
年龄在 18 岁及以上，经裁决风险评分的初级保健患者。
（1）从结构化访谈中提取主题和与提供者裁决行为相关的离散变量；（2）比较风险模型之间的一致性统计数据和敏感性。
2018 年，共有 47940 名患者由 PCP 进行了裁决。访谈结果表明，在裁决中，提供者会考虑疾病严重程度、自我管理技能、行为健康以及风险评分是否可操作。与算法风险评分相比，提供者上调评分与患者男性（比值比[OR]1.24，95%置信区间[CI]1.15-1.34）、年龄>65 岁（OR 2.55，CI 2.24-2.91）、黑人（OR 1.26，CI 1.02-1.55）、服用 10 种以上药物（OR 4.87，CI 4.27-5.56）、抑郁筛查阳性（OR 1.57，CI 1.43-1.72）和血红蛋白 A1c>9（OR 1.89，CI 1.52-2.33）显著相关。总体而言，与商业算法相比，经裁决的风险模型在所有结局上表现更好：急诊科就诊（c 统计量 0.689 与 0.684，p<0.01）、住院治疗（c 统计量 0.663 与 0.649，p<0.01）和死亡（c 统计量 0.753 与 0.721，p<0.01）。当仅限于男性或老年人时，与商业模型相比，裁决模型显示出改进或非劣效性的性能。
风险分层的提供者裁决可提高模型性能，因为提供者对其患者有个人的了解，并能够将其培训应用于临床决策。
