Can early re-admission rates accurately detect poor-quality hospitals?

There is widespread interest in using external quality measures, such as early re-admission rates (ERRs), to evaluate hospital quality. To evaluate the feasibility of using ERRs to identify poor-quality hospitals, a Monte Carlo simulation model was developed that describes the predictive power of ERRs for the 190 hospitals in Michigan using different assumptions concerning the distribution and variability of quality problems, the number of years of data aggregated, and unmeasured case-mix differences. The ability of ERRs to distinguish 171 average-quality hospitals from 19 poor-quality hospitals (assigned to have 5% vs. 15% premature discharges) was evaluated. First, the largest diagnosis-related groups (DRGs) were studied to determine if they included cardiac, gastrointestinal, pulmonary, and neurologic diseases. Despite using the highly optimistic assumptions that premature discharges are readmitted 50% more frequently than appropriately timed discharges and that no ERR variation was caused by unmeasured case-mix differences between hospitals, the results were poor. For example, for DRG 127 (heart failure), high ERR outlier status (using a .05 probability cutoff) had a positive predictive value of only 36%, meaning that approximately two thirds of hospitals labeled "poor-quality" (high ERR outliers) were false-positive results. Next, we repeated the simulation with sample sizes aggregated for all medical DRGs. The positive predictive value was 72%, but was very sensitive to ERR variability due to non-quality-related factors (e.g., unmeasured case mix). The positive predictive value decreases to 45% if unmeasured case mix accounts for even 10% of observed hospital ERR variation. The circumstances under which DRG-specific ERRs would be useful to detect poor-quality hospitals are unlikely to occur. Even collapsing to all medical DRGs, ERRs are likely to be accurate predictors only if quality differences are quite large and if unmeasured case-mix differences account for a small amount of interhospital variation in ERRs.