Examining publication bias-a simulation-based evaluation of statistical tests on publication bias.

BACKGROUND

Publication bias is a form of scientific misconduct. It threatens the validity of research results and the credibility of science. Although several tests on publication bias exist, no in-depth evaluations are available that examine which test performs best for different research settings.

METHODS

Four tests on publication bias, Egger's test (FAT), p-uniform, the test of excess significance (TES), as well as the caliper test, were evaluated in a Monte Carlo simulation. Two different types of publication bias and its degree (0%, 50%, 100%) were simulated. The type of publication bias was defined either as , meaning the repeated analysis of new datasets, or , meaning the inclusion of covariates in order to obtain a significant result. In addition, the underlying effect ( = 0, 0.5, 1, 1.5), effect heterogeneity, the number of observations in the simulated primary studies ( = 100, 500), and the number of observations for the publication bias tests ( = 100, 1,000) were varied.

RESULTS

All tests evaluated were able to identify publication bias both in the and condition. The false positive rates were, with the exception of the 15%- and 20%-caliper test, unbiased. The FAT had the largest statistical power in the conditions, whereas under the TES was, except under effect heterogeneity, slightly better. The CTs were, however, inferior to the other tests under effect homogeneity and had a decent statistical power only in conditions with 1,000 primary studies.

DISCUSSION

The FAT is recommended as a test for publication bias in standard meta-analyses with no or only small effect heterogeneity. If two-sided publication bias is suspected as well as under the TES is the first alternative to the FAT. The 5%-caliper test is recommended under conditions of effect heterogeneity and a large number of primary studies, which may be found if publication bias is examined in a discipline-wide setting when primary studies cover different research problems.