Bonett Stephen, Lin Willey, Sexton Topper Patrina, Wolfe James, Golinkoff Jesse, Deshpande Aayushi, Villarruel Antonia, Bauermeister José
School of Nursing, University of Pennsylvania, Philadelphia, PA, United States.
Department of Psychology, Ashoka University, Sonepat, India.
JMIR Form Res. 2024 Jan 12;8:e47091. doi: 10.2196/47091.
Web-based surveys increase access to study participation and improve opportunities to reach diverse populations. However, web-based surveys are vulnerable to data quality threats, including fraudulent entries from automated bots and duplicative submissions. Widely used proprietary tools to identify fraud offer little transparency about the methods used, effectiveness, or representativeness of resulting data sets. Robust, reproducible, and context-specific methods of accurately detecting fraudulent responses are needed to ensure integrity and maximize the value of web-based survey research.
This study aims to describe a multilayered fraud detection system implemented in a large web-based survey about COVID-19 attitudes, beliefs, and behaviors; examine the agreement between this fraud detection system and a proprietary fraud detection system; and compare the resulting study samples from each of the 2 fraud detection methods.
The PhillyCEAL Common Survey is a cross-sectional web-based survey that remotely enrolled residents ages 13 years and older to assess how the COVID-19 pandemic impacted individuals, neighborhoods, and communities in Philadelphia, Pennsylvania. Two fraud detection methods are described and compared: (1) a multilayer fraud detection strategy developed by the research team that combined automated validation of response data and real-time verification of study entries by study personnel and (2) the proprietary fraud detection system used by the Qualtrics (Qualtrics) survey platform. Descriptive statistics were computed for the full sample and for responses classified as valid by 2 different fraud detection methods, and classification tables were created to assess agreement between the methods. The impact of fraud detection methods on the distribution of vaccine confidence by racial or ethnic group was assessed.
Of 7950 completed surveys, our multilayer fraud detection system identified 3228 (40.60%) cases as valid, while the Qualtrics fraud detection system identified 4389 (55.21%) cases as valid. The 2 methods showed only "fair" or "minimal" agreement in their classifications (κ=0.25; 95% CI 0.23-0.27). The choice of fraud detection method impacted the distribution of vaccine confidence by racial or ethnic group.
The selection of a fraud detection method can affect the study's sample composition. The findings of this study, while not conclusive, suggest that a multilayered approach to fraud detection that includes conservative use of automated fraud detection and integration of human review of entries tailored to the study's specific context and its participants may be warranted for future survey research.
基于网络的调查增加了参与研究的机会，并改善了接触不同人群的可能性。然而，基于网络的调查容易受到数据质量威胁，包括自动机器人的欺诈性条目和重复提交。广泛使用的用于识别欺诈的专有工具几乎没有提供关于所使用方法、有效性或所得数据集代表性的透明度。需要稳健、可重复且针对具体情况的准确检测欺诈性回复的方法，以确保基于网络的调查研究的完整性并最大化其价值。
本研究旨在描述在一项关于新冠疫情态度、信念和行为的大型基于网络的调查中实施的多层欺诈检测系统；检验该欺诈检测系统与一个专有欺诈检测系统之间的一致性；并比较两种欺诈检测方法各自所得的研究样本。
费城社区参与与评估共同调查是一项基于网络的横断面调查，远程招募13岁及以上居民，以评估新冠疫情如何影响宾夕法尼亚州费城的个人、社区和邻里。描述并比较了两种欺诈检测方法：（1）研究团队开发的多层欺诈检测策略，该策略结合了对回复数据的自动验证以及研究人员对研究条目的实时核查；（2）Qualtrics（Qualtrics）调查平台使用的专有欺诈检测系统。计算了整个样本以及被两种不同欺诈检测方法分类为有效的回复的描述性统计数据，并创建了分类表以评估两种方法之间的一致性。评估了欺诈检测方法对按种族或族裔划分的疫苗信心分布的影响。
在7950份完成的调查中，我们的多层欺诈检测系统将3228例（40.60%）判定为有效，而Qualtrics欺诈检测系统将4389例（55.21%）判定为有效。两种方法在分类上仅显示出“一般”或“极小”的一致性（κ = 0.25；95%置信区间0.23 - 0.27）。欺诈检测方法的选择影响了按种族或族裔划分的疫苗信心分布。
欺诈检测方法的选择会影响研究的样本构成。本研究的结果虽不具有决定性，但表明对于未来的调查研究，可能有必要采用多层欺诈检测方法，包括谨慎使用自动欺诈检测以及结合根据研究的具体情况及其参与者量身定制的人工条目审核。
