Department of Surgery, University of Michigan Health System, Ann Arbor.
Department of Surgery, Michigan State University College of Human Medicine, East Lansing.
JAMA Surg. 2014 May;149(5):467-74. doi: 10.1001/jamasurg.2013.4249.
Quality improvement platforms commonly use risk-adjusted morbidity and mortality to profile hospital performance. However, given small hospital caseloads and low event rates for some procedures, it is unclear whether these outcomes reliably reflect hospital performance.
To determine the reliability of risk-adjusted morbidity and mortality for hospital performance profiling using clinical registry data.
DESIGN, SETTING, AND PARTICIPANTS: A retrospective cohort study was conducted using data from the American College of Surgeons National Surgical Quality Improvement Program, 2009. Participants included all patients (N = 55,466) who underwent colon resection, pancreatic resection, laparoscopic gastric bypass, ventral hernia repair, abdominal aortic aneurysm repair, and lower extremity bypass.
Outcomes included risk-adjusted overall morbidity, severe morbidity, and mortality. We assessed reliability (0-1 scale: 0, completely unreliable; and 1, perfectly reliable) for all 3 outcomes. We also quantified the number of hospitals meeting minimum acceptable reliability thresholds (>0.70, good reliability; and >0.50, fair reliability) for each outcome.
For overall morbidity, the most common outcome studied, the mean reliability depended on sample size (ie, how high the hospital caseload was) and the event rate (ie, how frequently the outcome occurred). For example, mean reliability for overall morbidity was low for abdominal aortic aneurysm repair (reliability, 0.29; sample size, 25 cases per year; and event rate, 18.3%). In contrast, mean reliability for overall morbidity was higher for colon resection (reliability, 0.61; sample size, 114 cases per year; and event rate, 26.8%). Colon resection (37.7% of hospitals), pancreatic resection (7.1% of hospitals), and laparoscopic gastric bypass (11.5% of hospitals) were the only procedures for which any hospitals met a reliability threshold of 0.70 for overall morbidity. Because severe morbidity and mortality are less frequent outcomes, their mean reliability was lower, and even fewer hospitals met the thresholds for minimum reliability.
Most commonly reported outcome measures have low reliability for differentiating hospital performance. This is especially important for clinical registries that sample rather than collect 100% of cases, which can limit hospital case accrual. Eliminating sampling to achieve the highest possible caseloads, adjusting for reliability, and using advanced modeling strategies (eg, hierarchical modeling) are necessary for clinical registries to increase their benchmarking reliability.
质量改进平台通常使用风险调整发病率和死亡率来分析医院的绩效。然而，由于某些手术的医院病例量较小且事件发生率较低，因此尚不清楚这些结果是否可靠地反映了医院的绩效。
使用临床登记数据确定风险调整发病率和死亡率用于医院绩效分析的可靠性。
设计、设置和参与者：这是一项回顾性队列研究，使用了美国外科医师学会全国手术质量改进计划（2009 年）的数据。参与者包括所有接受结肠切除术、胰腺切除术、腹腔镜胃旁路手术、腹疝修复术、腹主动脉瘤修复术和下肢旁路手术的患者（N=55466）。
结果包括风险调整后的总发病率、严重发病率和死亡率。我们评估了所有 3 种结果的可靠性（0-1 量表：0，完全不可靠；1，完全可靠）。我们还量化了每个结果达到最低可接受可靠性阈值（>0.70，可靠；>0.50，尚可）的医院数量。
对于最常见的研究结果——总发病率，其平均可靠性取决于样本量（即医院的病例量有多高）和事件发生率（即结果发生的频率）。例如，腹主动脉瘤修复术的总发病率的平均可靠性较低（可靠性为 0.29；样本量为每年 25 例；事件发生率为 18.3%）。相比之下，结肠切除术的总发病率的平均可靠性较高（可靠性为 0.61；样本量为每年 114 例；事件发生率为 26.8%）。只有结肠切除术（37.7%的医院）、胰腺切除术（7.1%的医院）和腹腔镜胃旁路手术（11.5%的医院）的任何医院的总发病率可靠性都达到了 0.70 的可靠性阈值。由于严重发病率和死亡率是较不常见的结果，因此它们的平均可靠性较低，达到最低可靠性阈值的医院数量更少。
最常报告的结果衡量标准在区分医院绩效方面的可靠性较低。对于仅抽样而不是收集 100%病例的临床登记处来说，这尤其重要，这可能会限制医院的病例积累。为了提高基准测试的可靠性，临床登记处需要消除抽样以实现尽可能高的病例量、对可靠性进行调整以及使用高级建模策略（例如，分层建模）。
