Comparing hospital mortality in adult patients with pneumonia. A case study of statistical methods in a managed care program.

OBJECTIVE

To compare and contrast a managed care program's analysis of differences in hospital mortality with results obtained by accepted statistical methods.

DESIGN

A re-analysis of computerized discharge data using the same method used by a managed care program, and using conventional methods of categorical data analysis. One thousand computer simulations of a method for comparing hospitals by severity-adjusted mortality were done to determine the probability of falsely identifying hospitals as high-mortality outliers.

SETTING

22 acute care hospitals in central Pennsylvania.

PATIENTS

All adult patients with pneumonia (n = 4587; diagnosis-related groups 089-090) less than 65 years of age who were discharged from the 22 hospitals in 1989, 1990, and 1991, excluding patients with the acquired immunodeficiency syndrome and transplant recipients.

MEASUREMENTS

In-hospital mortality adjusted for age and severity of illness using MedisGroups admission severity group score.

RESULTS

The hospital that had the highest mortality for adult pneumonia according to the managed care program's analysis did not, according to an appropriate analysis, differ significantly from other area hospitals (likelihood ratio test, P = 0.23). Random variation in this sample of patients with a low average mortality rate (3.5%) showed a 60% chance that 1 or more of the 22 hospitals would be falsely identified as a "high-mortality outlier" when simplistic statistical methods were used.

CONCLUSION

Organizations seeking to compare the quality of hospitals and physicians through outcome data need to recognize that simplistic methods applicable to large samples fail when applied to the outcomes of typical patients, such as those admitted for pneumonia. Although these comparisons are much in demand, careful attention must be paid to their statistical methods to ensure validity and fairness.