Department of Experimental Psychology, University of Groningen, 9712 TS, Groningen, The Netherlands.
Department of Psychology, University of Amsterdam, 1018 XA, Amsterdam, The Netherlands.
Behav Res Methods. 2018 Aug;50(4):1614-1631. doi: 10.3758/s13428-018-1054-3.
Psychological experiments often yield data that are hierarchically structured. A number of popular shortcut strategies in cognitive modeling do not properly accommodate this structure and can result in biased conclusions. To gauge the severity of these biases, we conducted a simulation study for a two-group experiment. We first considered a modeling strategy that ignores the hierarchical data structure. In line with theoretical results, our simulations showed that Bayesian and frequentist methods that rely on this strategy are biased towards the null hypothesis. Secondly, we considered a modeling strategy that takes a two-step approach by first obtaining participant-level estimates from a hierarchical cognitive model and subsequently using these estimates in a follow-up statistical test. Methods that rely on this strategy are biased towards the alternative hypothesis. Only hierarchical models of the multilevel data lead to correct conclusions. Our results are particularly relevant for the use of hierarchical Bayesian parameter estimates in cognitive modeling.
心理实验通常会产生层次结构的数据。认知建模中的一些流行的快捷策略并没有很好地适应这种结构，可能会导致有偏差的结论。为了评估这些偏差的严重程度，我们对一个两组实验进行了模拟研究。我们首先考虑了一种忽略层次数据结构的建模策略。模拟结果与理论结果一致，表明依赖这种策略的贝叶斯和频率方法偏向于零假设。其次，我们考虑了一种两步建模策略，首先从层次认知模型中获得参与者水平的估计，然后在后续的统计测试中使用这些估计。依赖这种策略的方法偏向于替代假设。只有对多层次数据的层次模型才能得出正确的结论。我们的结果对认知建模中层次贝叶斯参数估计的使用特别重要。
