Identification of preventable trauma deaths: confounded inquiries?

The published evaluation of methods for identifying preventable trauma deaths contains many unstudied confounding factors. To investigate the reliability of methods for identifying such preventable deaths, we compared three consensus systems using separate five-member general review panels assessing 20 non-central nervous system fatalities: panel A, independent judgments; panel B, discussion of all cases preceding individual judgments; and panel C, independent judgments followed by discussion and equivocal case reassignment. The Kappa concordance index was low for all methods (method A, 0.20; methods B and C, 0.40). Of the 11 deaths judged preventable by at least one panel, only one death was judged preventable by all three panels. Consensus agreement (four of five assessors) was 20% for panel A, 45% for panel B, and 10% for panel C (difference between panels B and C, p less than 0.03). In panel C, discussion affected the rate of equivocal case designation from 30% to 5%. Thus different consensus methods yielded different results. We conclude that individual case review can be severely flawed and therefore should not be used to measure institutional quality of patient care. We recommend that assessment of institutional performance should be based on objective evaluation methods, which require the study of patient population outcomes, rather than on subjective methods in which individual cases are reviewed.