Kadlec Jan, Walsh Catherine R, Sadé Uri, Amir Ariel, Rissman Jesse, Ramot Michal
Department of Brain Sciences, Weizmann Institute of Science, Rehovot, Israel.
Department of Psychology, University of California, Los Angeles, CA, USA.
Commun Psychol. 2024 Jul 4;2(1):64. doi: 10.1038/s44271-024-00114-4.
Surging interest in individual differences has faced setbacks in light of recent replication crises in psychology, for example in brain-wide association studies exploring brain-behavior correlations. A crucial component of replicability for individual differences studies, which is often assumed but not directly tested, is the reliability of the measures we use. Here, we evaluate the reliability of different cognitive tasks on a dataset with over 250 participants, who each completed a multi-day task battery. We show how reliability improves as a function of number of trials, and describe the convergence of the reliability curves for the different tasks, allowing us to score tasks according to their suitability for studies of individual differences. We further show the effect on reliability of measuring over multiple time points, with tasks assessing different cognitive domains being differentially affected. Data collected over more than one session may be required to achieve trait-like stability.
鉴于心理学最近出现的复制危机，例如在探索大脑与行为相关性的全脑关联研究中，对个体差异的浓厚兴趣遭遇了挫折。个体差异研究可重复性的一个关键要素，常常是被假定而非直接检验的，就是我们所使用测量方法的可靠性。在此，我们在一个有超过250名参与者的数据集上评估不同认知任务的可靠性，这些参与者每人都完成了一个为期多天的任务组。我们展示了可靠性如何随着试验次数的增加而提高，并描述了不同任务可靠性曲线的收敛情况，这使我们能够根据任务对个体差异研究的适用性对其进行评分。我们进一步展示了在多个时间点进行测量对可靠性的影响，评估不同认知领域的任务受到的影响各不相同。可能需要收集超过一个阶段的数据才能实现类似特质的稳定性。
