Department of Computer Science, Stanford University School of Engineering, Stanford, California, USA
Stanford Center for Biomedical Informatics Research, Stanford University School of Medicine, Stanford, California, USA.
BMJ Health Care Inform. 2022 Apr;29(1). doi: 10.1136/bmjhci-2021-100460.
The American College of Cardiology and the American Heart Association guidelines on primary prevention of atherosclerotic cardiovascular disease (ASCVD) recommend using 10-year ASCVD risk estimation models to initiate statin treatment. For guideline-concordant decision-making, risk estimates need to be calibrated. However, existing models are often miscalibrated for race, ethnicity and sex based subgroups. This study evaluates two algorithmic fairness approaches to adjust the risk estimators (group recalibration and equalised odds) for their compatibility with the assumptions underpinning the guidelines' decision rules.MethodsUsing an updated pooled cohorts data set, we derive unconstrained, group-recalibrated and equalised odds-constrained versions of the 10-year ASCVD risk estimators, and compare their calibration at guideline-concordant decision thresholds.
We find that, compared with the unconstrained model, group-recalibration improves calibration at one of the relevant thresholds for each group, but exacerbates differences in false positive and false negative rates between groups. An equalised odds constraint, meant to equalise error rates across groups, does so by miscalibrating the model overall and at relevant decision thresholds.
Hence, because of induced miscalibration, decisions guided by risk estimators learned with an equalised odds fairness constraint are not concordant with existing guidelines. Conversely, recalibrating the model separately for each group can increase guideline compatibility, while increasing intergroup differences in error rates. As such, comparisons of error rates across groups can be misleading when guidelines recommend treating at fixed decision thresholds.
The illustrated tradeoffs between satisfying a fairness criterion and retaining guideline compatibility underscore the need to evaluate models in the context of downstream interventions.
美国心脏病学会和美国心脏协会关于动脉粥样硬化性心血管疾病（ASCVD）一级预防的指南建议使用 10 年 ASCVD 风险评估模型来启动他汀类药物治疗。为了符合指南的决策，需要对风险估计值进行校准。然而，现有的模型往往在种族、民族和性别亚组中存在校准偏差。本研究评估了两种算法公平性方法，即对风险估计值进行分组重新校准（group recalibration）和均等机会校准（equalised odds），以使其与指南决策规则所依据的假设相匹配。
我们使用更新的汇总队列数据，推导出不受约束、分组重新校准和均等机会约束的 10 年 ASCVD 风险估计值，并比较它们在符合指南的决策阈值下的校准情况。
我们发现，与未受约束的模型相比，分组重新校准可以改善每个组的一个相关阈值下的校准，但会加剧组间假阳性和假阴性率的差异。均等机会约束旨在使各组的错误率均等化，但它会通过整体上和在相关决策阈值上的模型失准来实现这一目标。
因此，由于诱导的失准，由均等机会公平性约束学习的风险估计值指导的决策与现有指南不一致。相反，分别为每个组重新校准模型可以提高与指南的兼容性，同时增加组间错误率的差异。因此，当指南建议在固定决策阈值下进行治疗时，对组间错误率的比较可能会产生误导。
所说明的在满足公平标准和保留指南兼容性之间的权衡突显了在下游干预的背景下评估模型的必要性。
