Non-replication of association studies: "pseudo-failures" to replicate?

Recently, serious doubts have been cast on the usefulness of association studies as a means to genetically dissect complex diseases because most initial findings fail to replicate in subsequent studies. The reasons usually invoked are population stratification, genetic heterogeneity, and inflated Type I errors. In this article, we argue that, even when these problems are addressed, the scientific community usually has unreasonably high expectations on replication success, based on initial low P values, a phenomenon known as the replication fallacy. We present a modified formula that gives the replication power of a second association study based on the P value of an initial study. When both studies have similar sample sizes, this formula shows that: (1) a P value only slightly lower than the nominal alpha results in only approximately 50% replication power; (2) very low P values are required to achieve a replication power of at least 80% (e.g., at alpha = 0.05, a P value of <0.005 is required). Because many initially significant findings result in low replication power, replication failure should not be surprising or be interpreted as necessarily refuting the initial findings. We refer to replication failures for which the replication power is low as "pseudo-failures."