Spiess Martin, Jordan Pascal
Institute of Psychology, Department of Psychology and Human Movement Science, University of Hamburg, Hamburg, Germany.
Front Psychol. 2023 Sep 21;14:1266447. doi: 10.3389/fpsyg.2023.1266447. eCollection 2023.
Despite discussions about the replicability of findings in psychological research, two issues have been largely ignored: selection mechanisms and model assumptions. Both topics address the same fundamental question: Does the chosen statistical analysis tool adequately model the data generation process? In this article, we address both issues and show, in a first step, that in the face of selective samples and contrary to common practice, the validity of inferences, even when based on experimental designs, can be claimed without further justification and adaptation of standard methods only in very specific situations. We then broaden our perspective to discuss consequences of violated assumptions in linear models in the context of psychological research in general and in generalized linear mixed models as used in item response theory. These types of misspecification are oftentimes ignored in the psychological research literature. It is emphasized that the above problems cannot be overcome by strategies such as preregistration, large samples, replications, or a ban on testing null hypotheses. To avoid biased conclusions, we briefly discuss tools such as model diagnostics, statistical methods to compensate for selectivity and semi- or non-parametric estimation. At a more fundamental level, however, a twofold strategy seems indispensable: (1) iterative, cumulative theory development based on statistical methods with theoretically justified assumptions, and (2) empirical research on variables that affect (self-) selection into the observed part of the sample and the use of this information to compensate for selectivity.
尽管围绕心理学研究结果的可重复性展开了诸多讨论，但有两个问题在很大程度上被忽视了：选择机制和模型假设。这两个主题都涉及同一个基本问题：所选的统计分析工具是否能充分模拟数据生成过程？在本文中，我们将探讨这两个问题，并首先表明，面对选择性样本，与常见做法相反，即使基于实验设计，只有在非常特殊的情况下，才可以在无需进一步论证以及对标准方法进行调整的情况下宣称推理的有效性。然后，我们拓宽视野，在一般心理学研究的背景下，以及在项目反应理论中使用的广义线性混合模型中，讨论线性模型中假设被违反的后果。这类模型设定错误在心理学研究文献中常常被忽视。需要强调的是，诸如预先注册、大样本、重复研究或禁止检验零假设等策略无法克服上述问题。为避免得出有偏差的结论，我们简要讨论了诸如模型诊断、补偿选择性的统计方法以及半参数或非参数估计等工具。然而，在更基本的层面上，一种双重策略似乎必不可少：（1）基于具有理论依据假设的统计方法进行迭代式、累积式理论发展，以及（2）对影响样本观察部分的（自我）选择的变量进行实证研究，并利用这些信息来补偿选择性。
