Department of Anesthesiology and Perioperative Medicine, University of Rochester School of Medicine, Rochester, New York.
Department of Public Health Sciences, University of Rochester School of Medicine, Rochester, New York.
JAMA Netw Open. 2021 Dec 1;4(12):e2137647. doi: 10.1001/jamanetworkopen.2021.37647.
Unreliable performance measures can mask poor-quality care and distort financial incentives in value-based purchasing.
To examine the association between test-retest reliability and the reproducibility of hospital rankings.
DESIGN, SETTING, AND PARTICIPANTS: In a cross-sectional design, Centers for Medicare & Medicaid Services Hospital Compare data were analyzed for the 2017 (based on 2014-2017 data) and 2018 (based on 2015-2018 data) reporting periods. The study was conducted from December 13, 2020, to September 30, 2021. This analysis was based on 28 measures, including mortality (acute myocardial infarction, congestive heart failure, pneumonia, and coronary artery bypass grafting), readmissions (acute myocardial infarction, congestive heart failure, pneumonia, and coronary artery bypass grafting), and surgical complications (postoperative acute kidney failure, postoperative respiratory failure, postoperative sepsis, and failure to rescue).
Measure reliability based on test-retest reliability testing.
The reproducibility of hospital rankings was quantified by calculating the reclassification rate across the 2017 and 2018 reporting periods after categorizing the hospitals into terciles, quartiles, deciles, and statistical outliers. Linear regression analysis was used to examine the association between the reclassification rate and the intraclass correlation coefficient for each of the classification systems.
The analytic cohort consisted of 28 measures from 4452 hospitals with a median of 2927 (IQR, 2378-3160) hospitals contributing data for each measure. The hospitals participating in the Inpatient Prospective Payment System (n = 3195) had a median bed size of 141 (IQR, 69-261), average daily census of 70 (IQR, 24-155) patients, and a median disproportionate share hospital percentage of 38.2% (IQR, 18.7%-36.6%). The median intraclass correlation coefficient was 0.78 (IQR, 0.72-0.81), ranging between 0.50 and 0.85. The median reclassification rate was 70% (IQR, 62%-71%) when hospitals were ranked by deciles, 43% (IQR, 39%-45%) when ranked by quartiles, 34% (IQR, 31%-36%) when ranked by terciles, and 3.8% (IQR, 2.0%-6.2%) when ranked by outlier status. Increases in measure reliability were not associated with decreases in the reclassification rate. Each 0.1-point increase in the intraclass correlation coefficient was associated with a 6.80 (95% CI, 2.28-11.30; P = .005) percentage-point increase in the reclassification rate when hospitals were ranked into performance deciles, 4.15 (95% CI, 1.16-7.14; P = .008) when ranked into performance quartiles, 1.47 (95% CI, 1.84, 4.77; P = .37) when ranked into performance terciles, and 3.70 (95% CI, 1.30-6.09; P = .004) when ranked by outlier status.
In this study, more reliable measures were not associated with lower rates of reclassifying hospitals using test-retest reliability testing. These findings suggest that measure reliability should not be assessed with test-retest reliability testing.
不可靠的绩效衡量标准可能掩盖质量差的护理，并扭曲基于价值的购买中的财务激励措施。
研究测试-重测可靠性与医院排名再现性之间的关联。
设计、地点和参与者：在一项横断面设计中，分析了 2017 年（基于 2014-2017 年的数据）和 2018 年（基于 2015-2018 年的数据）报告期的医疗保险和医疗补助服务中心医院比较数据。这项研究是从 2020 年 12 月 13 日至 2021 年 9 月 30 日进行的。本分析基于 28 项措施，包括死亡率（急性心肌梗死、充血性心力衰竭、肺炎和冠状动脉旁路移植术）、再入院率（急性心肌梗死、充血性心力衰竭、肺炎和冠状动脉旁路移植术）和手术并发症（术后急性肾衰竭、术后呼吸衰竭、术后败血症和抢救失败）。
基于测试-重测可靠性测试的衡量可靠性。
通过将医院分为三分位数、四分位数、十分位数和统计异常值，在 2017 年和 2018 年报告期内重新分类医院，量化了医院排名的再现性。线性回归分析用于检查每个分类系统的再分类率与组内相关系数之间的关系。
分析队列包括来自 4452 家医院的 28 项措施，中位数为 2927 家（IQR，2378-3160 家）医院为每项措施提供数据。参与住院病人预付款制度（n=3195）的医院平均床位规模为 141（IQR，69-261），平均每日普查 70（IQR，24-155）名患者，不成比例的医院比例中位数为 38.2%（IQR，18.7%-36.6%）。中位数组内相关系数为 0.78（IQR，0.72-0.81），范围在 0.50 至 0.85 之间。中位数再分类率为 70%（IQR，62%-71%），当按十分位数对医院进行排名时，43%（IQR，39%-45%），当按四分位数排名时，34%（IQR，31%-36%），当按三分位数排名时，3.8%（IQR，2.0%-6.2%），当按异常值状态排名时。测量可靠性的提高与再分类率的降低无关。当医院按绩效十分位数排名时，组内相关系数每增加 0.1 分，再分类率就会增加 6.80（95%CI，2.28-11.30；P=0.005），按绩效四分位数排名时，再分类率增加 4.15（95%CI，1.16-7.14；P=0.008），按绩效三分位数排名时，再分类率增加 1.47（95%CI，1.84，4.77；P=0.37），按异常值状态排名时，再分类率增加 3.70（95%CI，1.30-6.09；P=0.004）。
在这项研究中，使用测试-重测可靠性测试衡量的更可靠的措施与使用测试-重测可靠性测试重新分类医院的比率降低无关。这些发现表明，不应使用测试-重测可靠性测试来评估衡量标准的可靠性。
