Ba Yang, Mancenido Michelle V, Chiou Erin K, Pan Rong
Ira A. Fulton Schools of Engineering, School of Computing and Augmented Intelligence, Data Science, Analytics and Engineering, Arizona State University, Suite 342AE, 3rd floor 699 S. Mill Avenue, 85281, Tempe, AZ, USA.
School of Mathematical and Natural Sciences, Arizona State University, Tempe, AZ, USA.
Behav Res Methods. 2025 Aug 8;57(9):251. doi: 10.3758/s13428-025-02757-5.
As crowdsourcing emerges as an efficient and cost-effective method for obtaining labels for machine learning datasets, it is important to assess the quality of crowd-provided data to improve analysis performance and reduce biases in subsequent machine learning tasks. Given the lack of ground truth in most cases of crowdsourcing, we refer to data quality as the annotators' consistency and credibility. Unlike the simple scenarios where kappa coefficient and intraclass correlation coefficient usually can apply, online crowdsourcing requires dealing with more complex situations. We introduce a systematic method for evaluating data quality and detecting spamming threats via variance decomposition, and we classify spammers into three categories based on their different behavioral patterns. A spammer index is proposed to assess entire data consistency, and two metrics are developed to measure crowd workers' credibility by utilizing the Markov chain and generalized random effects models. Furthermore, we demonstrate the practicality of our techniques and their advantages by applying them to a face verification task using both simulated and real-world data collected from two crowdsourcing platforms.
随着众包成为一种为机器学习数据集获取标签的高效且经济高效的方法，评估众包数据的质量对于提高分析性能和减少后续机器学习任务中的偏差至关重要。鉴于在大多数众包情况下缺乏地面真值，我们将数据质量定义为注释者的一致性和可信度。与通常可以应用卡帕系数和类内相关系数的简单场景不同，在线众包需要处理更复杂的情况。我们引入了一种通过方差分解评估数据质量和检测垃圾邮件威胁的系统方法，并根据垃圾邮件发送者的不同行为模式将其分为三类。提出了一个垃圾邮件发送者指数来评估整个数据的一致性，并开发了两个指标来利用马尔可夫链和广义随机效应模型来衡量众包工作者的可信度。此外，我们通过将我们的技术应用于使用从两个人工众包平台收集的模拟数据和真实世界数据的面部验证任务，证明了我们技术的实用性及其优势。
