Department of Behavioural Biology, University of Münster, Münster, Germany.
Otto Creutzfeldt Center for Cognitive and Behavioral Neuroscience, University of Münster, Münster, Germany.
PLoS Biol. 2022 May 5;20(5):e3001564. doi: 10.1371/journal.pbio.3001564. eCollection 2022 May.
The credibility of scientific research has been seriously questioned by the widely claimed "reproducibility crisis". In light of this crisis, there is a growing awareness that the rigorous standardisation of experimental conditions may contribute to poor reproducibility of animal studies. Instead, systematic heterogenisation has been proposed as a tool to enhance reproducibility, but a real-life test across multiple independent laboratories is still pending. The aim of this study was therefore to test whether heterogenisation of experimental conditions by using multiple experimenters improves the reproducibility of research findings compared to standardised conditions with only one experimenter. To this end, we replicated the same animal experiment in 3 independent laboratories, each employing both a heterogenised and a standardised design. Whereas in the standardised design, all animals were tested by a single experimenter; in the heterogenised design, 3 different experimenters were involved in testing the animals. In contrast to our expectation, the inclusion of multiple experimenters in the heterogenised design did not improve the reproducibility of the results across the 3 laboratories. Interestingly, however, a variance component analysis indicated that the variation introduced by the different experimenters was not as high as the variation introduced by the laboratories, probably explaining why this heterogenisation strategy did not bring the anticipated success. Even more interestingly, for the majority of outcome measures, the remaining residual variation was identified as an important source of variance accounting for 41% (CI95 [34%, 49%]) to 72% (CI95 [58%, 88%]) of the observed total variance. Despite some uncertainty surrounding the estimated numbers, these findings argue for systematically including biological variation rather than eliminating it in animal studies and call for future research on effective improvement strategies.
科学研究的可信度受到了广泛声称的“可重复性危机”的严重质疑。鉴于这一危机，人们越来越意识到，严格规范实验条件可能导致动物研究的可重复性较差。相反，系统的异质性已被提议作为提高可重复性的一种工具，但在多个独立实验室进行实际测试仍有待进行。因此，本研究旨在测试通过使用多个实验员来使实验条件异质化是否比只有一个实验员的标准化条件更能提高研究结果的可重复性。为此，我们在 3 个独立的实验室中复制了相同的动物实验，每个实验室都采用了异质化和标准化的设计。在标准化设计中，所有动物都由一名实验员进行测试；在异质化设计中，有 3 名不同的实验员参与了对动物的测试。与我们的预期相反，在异质化设计中包含多个实验员并没有提高结果在 3 个实验室中的可重复性。然而，有趣的是，方差分量分析表明，不同实验员引入的变异并不像实验室引入的变异那么高，这可能解释了为什么这种异质化策略没有带来预期的成功。更有趣的是，对于大多数结果测量指标，剩余的残差变异被确定为一个重要的变异来源，占观察到的总变异的 41%（95%CI[34%，49%]）至 72%（95%CI[58%，88%]）。尽管估计数字存在一些不确定性，但这些发现表明，在动物研究中，系统地包括生物变异而不是消除它是有必要的，并呼吁未来研究有效的改进策略。
