Reaction-time task reliability is more accurately computed with permutation-based split-half correlations than with Cronbach's alpha.

While it has become standard practice to report the reliability of self-report scales, it remains uncommon to do the same for experimental paradigms. To facilitate this practice, we review old and new ways to compute reliability in reaction-time tasks, and we compare their accuracy using a simulation study. Highly inaccurate and negatively biased reliability estimates are obtained through the common practice of averaging sets of trials and submitting them to Cronbach's alpha. Much more accurate reliability estimates are obtained using split-half reliability methods, especially by computing many random split-half correlations and aggregating them in a metric known as permutation-based split-half reliability. Through reanalysis of existing data and comparison of reliability values reported in the literature, we confirm that Cronbach's alpha also tends to be lower than split-half reliability in real data. We further establish a set of practices to maximize the accuracy of the permutation-based split-half reliability coefficient through simulations. We find that its accuracy is improved by ensuring each split-half dataset contains an approximately equal number of trials for each stimulus, by correcting the averaged correlation for test length using a modified variant of the Spearman-Brown formula, and by computing a sufficient number of split-half correlations: around 5,400 are needed to obtain a stable estimate for median-based double-difference scores computed from 30 participants and 256 trials. To conclude, we review the available software for computing this coefficient.