Price Matthew, Hidalgo Johanna E, Kim Julia N, Legrand Alison C, Brier Zoe M F, van Stolk-Cooke Katherine, Lansing Amy Hughes, Contractor Ateka A
Department of Psychological Science, University of Vermont, 2 Colchester Avenue, Burlington, Vermont, 05405, USA.
State University of New York Geneseo, Dept of Psychology, USA.
Comput Human Behav. 2024 Aug;157. doi: 10.1016/j.chb.2024.108253. Epub 2024 Apr 20.
Crowdsourcing is an essential data collection method for psychological research. Concerns about the validity and quality of crowdsourced data persist, however. A recent documented increase in the number of invalid responses within crowdsourced data has highlighted the need for quality control measures. Although a number of approaches are recommended, few have been empirically evaluated. The present study evaluated a Cyborg Method that used automated evaluation of participant meta-data and a review of short answer responses. Two samples were recruited - in the first, the Cyborg Method was applied after data collection to gauge the extent to which invalid responses were collected when quality controls were absent. In the second, the Cyborg Method was applied during data collection to determine if the method would proactively screen invalid responses. Results suggested that Cyborg Method identified a substantial portion of invalid responses and both automated and human evaluation components was necessary. Furthermore, the Cyborg Method could be applied proactively to screen invalid responses and substantially reduced the per participant cost of data collection. These results suggest that the Cyborg Method is a promising means by which to collect high quality crowdsourced data.
众包是心理学研究中一种重要的数据收集方法。然而，对于众包数据的有效性和质量的担忧一直存在。最近有记录显示众包数据中无效回复的数量有所增加，这凸显了质量控制措施的必要性。尽管推荐了多种方法，但很少有方法经过实证评估。本研究评估了一种半机械人方法，该方法使用对参与者元数据的自动评估和对简答题回复的审查。招募了两个样本——在第一个样本中，在数据收集后应用半机械人方法，以衡量在没有质量控制的情况下收集到的无效回复的程度。在第二个样本中，在数据收集期间应用半机械人方法，以确定该方法是否会主动筛选无效回复。结果表明，半机械人方法识别出了很大一部分无效回复，并且自动评估和人工评估组件都是必要的。此外，半机械人方法可以主动应用于筛选无效回复，并大幅降低每位参与者的数据收集成本。这些结果表明，半机械人方法是一种有前景的收集高质量众包数据的手段。
