Complexities in power analysis: Quantifying uncertainties with a Bayesian-classical hybrid approach.

Power analysis serves as the gold standard for evaluating study feasibility and justifying sample size. However, mainstream power analysis is often oversimplified, poorly reflecting complex reality during data analysis. This article highlights the complexities inherent in power analysis, especially when uncertainties present in data analysis are realistically taken into account. We introduce a Bayesian-classical hybrid approach to power analysis, which formally incorporates three sources of uncertainty into power estimates: (a) epistemic uncertainty regarding the unknown values of the effect size of interest, (b) sampling variability, and (c) uncertainty due to model approximation (i.e., models fit data imperfectly; Box, 1979; MacCallum, 2003). To illustrate the nature of estimated power from the Bayesian-classical hybrid method, we juxtapose its power estimates with those obtained from traditional (i.e., classical or frequentist) and Bayesian approaches. We employ an example in lexical processing (e.g., Yap & Seow, 2014) to illustrate underlying concepts and provide accompanying R and Rcpp code for computing power via the Bayesian-classical hybrid method. In general, power estimates become more realistic and much more varied after uncertainties are incorporated into their computation. As such, sample sizes should be determined by assurance (i.e., the mean of the power distribution) and the extent of variability in power estimates (e.g., interval width between 20th and 80th percentiles of the power distribution). We discuss advantages and challenges of incorporating the three stated sources of uncertainty into power analysis and, more broadly, research design. Finally, we conclude with future research directions. (PsycINFO Database Record (c) 2019 APA, all rights reserved).