Bartneck Christoph, Duenser Andreas, Moltchanova Elena, Zawieska Karolina
University of Canterbury, Private Bag 4800, 8140 Christchurch, New Zealand.
CSIRO Computational Informatics Castray Esplanade, Hobart 7000, Tasmania, Australia.
PLoS One. 2015 Apr 14;10(4):e0121595. doi: 10.1371/journal.pone.0121595. eCollection 2015.
Computer and internet based questionnaires have become a standard tool in Human-Computer Interaction research and other related fields, such as psychology and sociology. Amazon's Mechanical Turk (AMT) service is a new method of recruiting participants and conducting certain types of experiments. This study compares whether participants recruited through AMT give different responses than participants recruited through an online forum or recruited directly on a university campus. Moreover, we compare whether a study conducted within AMT results in different responses compared to a study for which participants are recruited through AMT but which is conducted using an external online questionnaire service. The results of this study show that there is a statistical difference between results obtained from participants recruited through AMT compared to the results from the participant recruited on campus or through online forums. We do, however, argue that this difference is so small that it has no practical consequence. There was no significant difference between running the study within AMT compared to running it with an online questionnaire service. There was no significant difference between results obtained directly from within AMT compared to results obtained in the campus and online forum condition. This may suggest that AMT is a viable and economical option for recruiting participants and for conducting studies as setting up and running a study with AMT generally requires less effort and time compared to other frequently used methods. We discuss our findings as well as limitations of using AMT for empirical studies.
基于计算机和互联网的调查问卷已成为人机交互研究以及其他相关领域（如心理学和社会学）的标准工具。亚马逊的土耳其机器人（AMT）服务是一种招募参与者并进行某些类型实验的新方法。本研究比较了通过AMT招募的参与者与通过在线论坛招募或直接在大学校园招募的参与者给出的回答是否不同。此外，我们还比较了在AMT内进行的研究与通过AMT招募参与者但使用外部在线问卷服务进行的研究相比，是否会产生不同的回答。本研究结果表明，通过AMT招募的参与者所获得的结果与在校园或通过在线论坛招募的参与者所获得的结果之间存在统计学差异。然而，我们认为这种差异非常小，以至于没有实际影响。在AMT内进行研究与使用在线问卷服务进行研究之间没有显著差异。直接从AMT内获得的结果与在校园和在线论坛条件下获得的结果之间没有显著差异。这可能表明，AMT是招募参与者和进行研究的一种可行且经济的选择，因为与其他常用方法相比，使用AMT设置和进行一项研究通常需要更少的精力和时间。我们讨论了我们的研究结果以及将AMT用于实证研究的局限性。
