Litman Leib, Robinson Jonathan, Rosenzweig Cheskie
Department of Psychology, Lander College, 75-31 150th Street, Flushing, NY, 11367, USA,
Behav Res Methods. 2015 Jun;47(2):519-28. doi: 10.3758/s13428-014-0483-x.
In this study, we examined data quality among Amazon Mechanical Turk (MTurk) workers based in India, and the effect of monetary compensation on their data quality. Recent studies have shown that work quality is independent of compensation rates, and that compensation primarily affects the quantity but not the quality of work. However, the results of these studies were generally based on compensation rates below the minimum wage, and far below a level that was likely to play a practical role in the lives of workers. In this study, compensation rates were set around the minimum wage in India. To examine data quality, we developed the squared discrepancy procedure, which is a task-based quality assurance approach for survey tasks whose goal is to identify inattentive participants. We showed that data quality is directly affected by compensation rates for India-based participants. We also found that data were of a lesser quality among India-based than among US participants, even when optimal payment strategies were utilized. We additionally showed that the motivation of MTurk users has shifted, and that monetary compensation is now reported to be the primary reason for working on MTurk, among both US- and India-based workers. Overall, MTurk is a constantly evolving marketplace where multiple factors can contribute to data quality. High-quality survey data can be acquired on MTurk among India-based participants when an appropriate pay rate is provided and task-specific quality assurance procedures are utilized.
在本研究中，我们考察了印度地区亚马逊土耳其机器人（MTurk）众包工作者的数据质量，以及货币报酬对其数据质量的影响。近期研究表明，工作质量与报酬率无关，报酬主要影响工作数量而非质量。然而，这些研究结果通常基于低于最低工资的报酬率，且远低于可能在工作者生活中发挥实际作用的水平。在本研究中，报酬率设定在印度最低工资水平左右。为考察数据质量，我们开发了平方差异程序，这是一种基于任务的质量保证方法，用于调查任务，其目标是识别不专注的参与者。我们发现，印度参与者的数据质量直接受到报酬率的影响。我们还发现，即使采用了最优支付策略，印度参与者的数据质量仍低于美国参与者。我们还表明，MTurk用户的动机已经转变，现在货币报酬被报告为美国和印度工作者在MTurk上工作的主要原因。总体而言，MTurk是一个不断发展的市场，多种因素会影响数据质量。当提供适当的支付率并采用特定任务的质量保证程序时，在印度参与者中可以在MTurk上获得高质量的调查数据。
