Detecting Deception and Ensuring Data Integrity in a Nationwide mHealth Randomized Controlled Trial: Factorial Design Survey Study.

BACKGROUND

Social behavioral research studies have increasingly shifted to remote recruitment and enrollment procedures. This shifting landscape necessitates evolving best practices to help mitigate the negative impacts of deceptive attempts (eg, fake profiles and bots) at enrolling in behavioral research.

OBJECTIVE

This study aimed to develop and implement robust deception detection procedures during the enrollment period of a remotely conducted randomized controlled trial.

METHODS

A 32-group (2×2×2×2×2) factorial design study was conducted from November 2021 to September 2022 to identify mobile health (mHealth) survey design features associated with the highest completion rates of smartphone-based ecological momentary assessments (n=485). Participants were required to be at least 18 years old, live in the United States, and own an Android smartphone that was compatible with the Insight app that was used in the study. Recruitment was conducted remotely through Facebook advertisements, a 5-minute REDCap (Research Electronic Data Capture) prescreener, and a screening and enrollment phone call. The research team created and implemented a 12-step checklist (eg, address verification and texting a copy of picture identification) to identify and prevent potentially deceptive attempts to enroll in the study. Descriptive statistics were calculated to understand the prevalence of various types of deceptive attempts at study enrollment.

RESULTS

Facebook advertisements resulted in 5236 initiations of the REDCap prescreener. A digital deception detection procedure was implemented for those who were deemed pre-eligible (n=1928). This procedure resulted in 26% (501/1928) of prescreeners being flagged as potentially deceptive. Completing multiple prescreeners (301/501, 60.1%) and providing invalid addresses (156/501, 31.1%) were the most common reasons prescreeners were flagged. An additional 1% (18/1928) of prescreeners were flagged as potentially deceptive during the subsequent study screening and enrollment phone call. Reasons for exclusion at the screening and enrollment phone call level included having an invalid phone type (6/18, 33.3%), completing multiple prescreeners (6/18, 33.3%), and providing an invalid address (5/18, 27.7%). This resulted in 1409 individuals being eligible after all deception checks were completed. Postenrollment social security number checks revealed that 3 (0.6%) fully enrolled participants out of 485 provided erroneous social security numbers during the screening process.

CONCLUSIONS

Implementation of a deception detection procedure in a remotely conducted randomized controlled trial resulted in a substantial proportion of cases being flagged as potentially engaging in deceptive attempts at study enrollment. The results of the deception detection procedures in this study confirmed the need for vigilance in conducting remote behavioral research in order to maintain data integrity. Implementing systematic deception detection procedures may support study administration, data quality, and participant safety in remotely conducted behavioral research.

TRIAL REGISTRATION

ClinicalTrials.gov NCT05194228; https://clinicaltrials.gov/study/NCT05194228.