Researcher judgment and study design: challenges of using administrative data.

BACKGROUND

Questions have been raised about methods of studies finding substantial undercounting of workplace injuries and illnesses by the Bureau of Labor Statistics (BLS) and workers' compensation agencies. A more recent study of Minnesota concluded that the BLS survey captures 84-90% of workers' compensation cases.

METHODS

We examined the sensitivity of findings in two studies to alternate sample definitions and study assumptions.

RESULTS

Applying alternate sample construction rules to the earlier study increased estimated BLS reporting rates from 68% to 77%, assuming source independence. Applying alternate assumptions to the more recent Minnesota study reduced its high estimate of BLS reporting from 90% to 53-64%.

CONCLUSIONS

Studies linking administrative data from different sources require substantial judgment in constructing research datasets and choosing analytic methods. Moreover, different sample construction rules lead to different results. This suggests that sensitivity analysis should be carried out when alternatives cannot be ruled out. In this case, sensitivity analysis supports the hypothesis of substantial underreporting.