Department of Psychology, Northumbria University.
Perspect Psychol Sci. 2021 Mar;16(2):358-376. doi: 10.1177/1745691620964193. Epub 2021 Jan 5.
Heterogeneity emerges when multiple close or conceptual replications on the same subject produce results that vary more than expected from the sampling error. Here we argue that unexplained heterogeneity reflects a lack of coherence between the concepts applied and data observed and therefore a lack of understanding of the subject matter. Typical levels of heterogeneity thus offer a useful but neglected perspective on the levels of understanding achieved in psychological science. Focusing on continuous outcome variables, we surveyed heterogeneity in 150 meta-analyses from cognitive, organizational, and social psychology and 57 multiple close replications. Heterogeneity proved to be very high in meta-analyses, with powerful moderators being conspicuously absent. Population effects in the average meta-analysis vary from small to very large for reasons that are typically not understood. In contrast, heterogeneity was moderate in close replications. A newly identified relationship between heterogeneity and effect size allowed us to make predictions about expected heterogeneity levels. We discuss important implications for the formulation and evaluation of theories in psychology. On the basis of insights from the history and philosophy of science, we argue that the reduction of heterogeneity is important for progress in psychology and its practical applications, and we suggest changes to our collective research practice toward this end.
当对同一主题进行多次紧密或概念上的复制时，如果产生的结果比预期的抽样误差变化更大，则会出现异质性。在这里，我们认为无法解释的异质性反映了所应用的概念与观察到的数据之间缺乏一致性，因此也反映了对主题的理解不足。典型的异质水平因此为心理科学中所达到的理解水平提供了一个有用但被忽视的视角。我们关注连续的结果变量，对认知、组织和社会心理学的 150 项荟萃分析和 57 项紧密重复研究中的异质性进行了调查。荟萃分析中的异质性非常高，而强大的调节因素却明显缺失。由于通常无法理解的原因，平均荟萃分析中的群体效应从很小到非常大不等。相比之下，紧密重复中的异质性是适度的。我们发现异质性和效应量之间存在新的关系，这使我们能够对预期的异质水平进行预测。我们讨论了这对心理学中理论的制定和评估的重要意义。基于科学史和科学哲学的见解，我们认为减少异质性对于心理学及其实际应用的进展很重要，我们建议为此目的改变我们的集体研究实践。
