Significance Testing for Differences Between Baseline Variables Versus the I2 Test in Detecting Selection Bias in Randomised Controlled Trials: A Simulation Study.

AIM

The aim of the study is to test the null hypothesis that the specificities and sensitivities of the p-value-based significance test for differences between baseline variables and the I test for single trials do not significantly differ in detecting selection bias in randomised controlled trials (RCTs).

METHODS

In MS Excel (Microsoft Corp., Redmond, WA, US), 100 trials were simulated, each consisting of two treatment groups (A and B), with 100 subjects in each group. Fifty trials were biased, while 50 remained non-biased. Both tests were applied to all trials, yielding true positive, false positive, false negative, and true negative per test. Subsequently, sensitivities and specificities with a 95% confidence interval (CI) were calculated and statistically compared using the z-test.

RESULTS

No false positive results were observed, and subsequently, the specificities of both tests were identical (100.00%; 95% CI: 92.89%-100.00%). The sensitivity for the significance test and I test was 24.00% (95% CI: 13.06%-38.17%) and 76.00% (95% CI: 61.83%-86.94%), respectively. A statistical comparison of the test sensitivities yielded a significant result in favour of the I test (z = 5.2; p < 0.0001). Consequently, the null hypothesis for the tests' sensitivities was rejected.

CONCLUSION

The I test appears to be a more effective method than the p-value-based significance test for detecting selection bias in RCTs.