Survival-Inferred Fragility of Statistical Significance in Phase III Oncology Trials.

BACKGROUND

Statistical significance currently defines superiority in phase III oncology trials. However, this practice is increasingly questioned. Here, we estimated the fragility of phase III oncology trials.

METHODS

Using Kaplan-Meier curves for the primary endpoints of 230 two-arm superiority phase III oncology trials, we reconstructed data for individual patients. We estimated the survival-inferred fragility index (SIFI) by iteratively flipping the best responder from the experimental arm to the control arm (SIFI) until the interpretation was changed according to the significance threshold of each trial. Severe fragility was defined by SIFI ≤1%.

RESULTS

This study included 230 trials enrolling 184,752 patients. The median number of patients required to change trial interpretation was 8 (interquartile range, 4 to 19) or 1.4% (interquartile range, 0.7% to 3%) per SIFI. Estimations of SIFI by multiple methods were largely consistent. For trials with an overall survival primary endpoint, the median SIFI was 1% (IQR, 0.5% to 1.9%). Severe fragility was found in 87 trials (38%). As a continuous statistic, the original value-but not its binary significance interpretation-was associated with fragility and severe fragility. Trials with subsequent FDA approval had lower odds of severe fragility. Lastly, the underlying survival model had differential effects on SIFI estimation.

CONCLUSIONS

Even among phase III oncology trials, which directly inform patient care, changes in the outcomes of few patients are often sufficient to change statistical significance and trial interpretation. These findings imply that current definitions of statistical significance used in phase III oncology are inadequate to identify replicable findings.