Warwick Business School, University of Warwick, Coventry, CV4 7AL, UK.
Mathematica Policy Research, Ann Arbor, MI, 48103, USA; Research Center for Group Dynamics, University of Michigan, Ann Arbor, MI 48106, USA.
Trends Cogn Sci. 2017 Oct;21(10):736-748. doi: 10.1016/j.tics.2017.06.007. Epub 2017 Aug 10.
Crowdsourcing data collection from research participants recruited from online labor markets is now common in cognitive science. We review who is in the crowd and who can be reached by the average laboratory. We discuss reproducibility and review some recent methodological innovations for online experiments. We consider the design of research studies and arising ethical issues. We review how to code experiments for the web, what is known about video and audio presentation, and the measurement of reaction times. We close with comments about the high levels of experience of many participants and an emerging tragedy of the commons.
众包数据收集，即从在线劳动力市场招募的研究参与者，现在在认知科学中很常见。我们来回顾一下参与众包的人员以及普通实验室能够联系到的人员。我们讨论了可重复性并回顾了一些最近的在线实验方法创新。我们考虑了研究设计和出现的伦理问题。我们回顾了如何为网络编写实验代码，了解有关视频和音频演示的信息，以及反应时间的测量。最后，我们对许多参与者的丰富经验以及公共资源的悲剧进行了评论。
