Innovations Center, Truth Initiative, Washington, DC, United States.
Department of Medicine, Mayo Clinic College of Medicine and Science, Rochester, MN, United States.
J Med Internet Res. 2024 Nov 27;26:e54248. doi: 10.2196/54248.
Detecting early dropout from digital interventions is crucial for developing strategies to enhance user retention and improve health-related behavioral outcomes. Bricker and colleagues proposed a single metric that accurately predicted early dropout from 4 digital tobacco cessation interventions based on log-in data in the initial week after registration. Generalization of this method to additional interventions and modalities would strengthen confidence in the approach and facilitate additional research drawing on it to increase user retention.
This study had two research questions (RQ): RQ1-can the study by Bricker and colleagues be replicated using data from a large-scale observational, multimodal intervention to predict early dropout? and RQ2-can first-week engagement patterns identify users at the greatest risk for early dropout, to inform development of potential "rescue" interventions?
Data from web users were drawn from EX, a freely available, multimodal digital intervention for tobacco cessation (N=70,265). First-week engagement was operationalized as any website page views or SMS text message responses within 1 week after registration. Early dropout was defined as having no subsequent engagement after that initial week through 1 year. First, a multivariate regression model was used to predict early dropout. Model predictors were dichotomous measures of engagement in each of the initial 6 days (days 2-7) following registration (day 1). Next, 6 univariate regression models were compared in terms of their discrimination ability to predict early dropout. The sole predictor of each model was a dichotomous measure of whether users had reengaged with the intervention by a particular day of the first week (calculated separately for each of 2-7 days).
For RQ1, the area under the receiver operating characteristic curve (AUC) of the multivariate model in predicting dropout after 1 week was 0.72 (95% CI 0.71-0.73), which was within the range of AUC metrics found in the study by Bricker and colleagues. For RQ2, the AUCs of the univariate models increased with each successive day until day 4 (0.66, 95% CI 0.65-0.67). The sensitivity of the models decreased (range 0.79-0.59) and the specificity increased (range 0.48-0.73) with each successive day.
This study provides independent validation of the use of first-week engagement to predict early dropout, demonstrating that the method generalizes across intervention modalities and engagement metrics. As digital intervention researchers continue to address the challenges of low engagement and early dropout, these results suggest that first-week engagement is a useful construct with predictive validity that is robust across interventions and definitions. Future research should explore the applicability and efficiency of this model to develop interventions to increase retention and improve health behavioral outcomes.
及早发现数字干预措施的使用者流失对于制定策略以提高用户留存率和改善与健康相关的行为结果至关重要。Bricker 及其同事提出了一种单一指标，该指标可根据注册后第一周内的登录数据准确预测 4 项数字戒烟干预措施的早期使用者流失情况。将该方法推广到其他干预措施和模式中，将增强对该方法的信心，并促进利用该方法进行更多的研究，以提高用户留存率。
本研究有两个研究问题（RQ）：RQ1-是否可以使用大规模观察性、多模式干预措施的数据来复制 Bricker 及其同事的研究，以预测早期使用者流失？RQ2-第一周的参与模式是否可以识别出最有可能早期流失的用户，以告知潜在“挽救”干预措施的开发？
从 EX（一个免费的、多模式的戒烟数字干预措施）的网络用户中提取数据（N=70265）。第一周的参与度被定义为注册后 1 周内的任何网站页面浏览量或短信回复。早期流失被定义为在初始周后至 1 年内没有后续参与。首先，使用多元回归模型预测早期流失。模型预测因子是注册后第 2-7 天（第 1 天）中每日参与情况的二分测量值。接下来，根据预测早期流失的能力比较了 6 个单变量回归模型。每个模型的唯一预测因子是第 1 周内特定某一天用户是否重新参与干预措施的二分测量值（分别为第 2-7 天中的每一天计算）。
对于 RQ1，多变量模型预测 1 周后辍学的受试者工作特征曲线（AUC）下面积为 0.72（95%CI 0.71-0.73），这在 Bricker 及其同事的研究中发现的 AUC 指标范围内。对于 RQ2，单变量模型的 AUC 随着第 1 天的推移而增加，直到第 4 天（0.66，95%CI 0.65-0.67）。模型的灵敏度（范围为 0.79-0.59）随着第 1 天的推移而降低，特异性（范围为 0.48-0.73）随着第 1 天的推移而增加。
本研究对使用第一周的参与度来预测早期流失进行了独立验证，证明了该方法在干预措施和参与度指标上具有普遍性。随着数字干预研究人员继续应对低参与度和早期流失的挑战，这些结果表明，第一周的参与度是一种有用的具有预测有效性的构建，在各种干预措施和定义中都是稳健的。未来的研究应该探讨该模型的适用性和效率，以开发提高留存率和改善健康行为结果的干预措施。
