AAA benchmarking by Dr Foster: a cause for concern?

INTRODUCTION

The publication of interpretable performance data for hospitals is an important service. In November 2002, the medical benchmarking company Dr Foster published a league table based on the results of abdominal aortic aneurysm (AAA) repair. The purpose of our study was to establish the validity of the data used in benchmarking.

PATIENTS AND METHODS

Data on elective infra-renal AAA (IRAAA) repair was obtained from three sources. Data used by Dr Foster was based on the hospital PAS system. The databases for both Dr Foster and PAS were analysed and cross-referenced to the vascular unit database maintained by a separately employed audit co-ordinator.

RESULTS

Of 395 total aortic aneurysm repairs, 223 (56%) were identified as elective IRAAA repairs on the unit database. Of these, 125 were identified on the PAS database and 115 on the Dr Foster database. The number of deaths was the same in both the unit and Dr Foster databases (n = 11) but the Dr Foster database included deaths in patients who had undergone juxtarenal (n = 1), Type III TAAA (n = 2) and Type IV TAAA (n = 4) repairs and omitted 7 deaths following IRAAA. The sensitivity and specificity for the PAS dataset was 0.51 and 0.93, respectively. For Dr Foster, the results were worse with a sensitivity and specificity of 0.41 and 0.86, respectively. The accuracy of the data was 0.6 and 0.69 for Dr Foster and PAS, respectively. Standardised mortality ratios (SMRs) were used to rank hospitals. Dr Foster's published SMR for elective AAA repair for our unit was 160. The actual SMR was 67.

CONCLUSIONS

Robust and accurate published league tables should be supported and commended but currently available data appear to be misleading and may cause unnecessary concern to patients.