Black Dog Institute, The University of New South Wales Sydney, Randwick, Australia.
School of Psychiatry, The University of New South Wales Sydney, Randwick, Australia.
J Med Internet Res. 2021 Mar 11;23(3):e18348. doi: 10.2196/18348.
Randomized controlled trials (RCTs) with vigorous study designs are vital for determining the efficacy of treatments. Despite the high internal validity attributed to RCTs, external validity concerns limit the generalizability of results to the general population. Bias can be introduced, for example, when study participants who self-select into a trial are more motivated to comply with study conditions than are other individuals. These external validity considerations extend to e-mental health (eMH) research, especially when eMH tools are designed for public access and provide minimal or no supervision.
Clustering techniques were employed to identify engagement profiles of RCT participants and community users of a self-guided eMH program. This exploratory approach inspected actual, not theorized, RCT participant and community user engagement patterns. Both samples had access to the eMH program over the same time period and received identical usage recommendations on the eMH program website. The aim of this study is to help gauge expectations of similarities and differences in usage behaviors of an eMH tool across evaluation and naturalistic contexts.
Australian adults signed up to myCompass, a self-guided online treatment program created to reduce mild to moderate symptoms of negative emotions. They did so either by being part of an RCT onboarding (160/231, 69.6% female) or by accessing the program freely on the internet (5563/8391, 66.30% female) between October 2011 and October 2012. During registration, RCT participants and community users provided basic demographic information. Usage metrics (number of logins, trackings, and learning activities) were recorded by the system.
Samples at sign-up differed significantly in age (P=.003), with community users being on average 3 years older (mean 41.78, SD 13.64) than RCT participants (mean 38.79, SD 10.73). Furthermore, frequency of program use was higher for RCT participants on all usage metrics compared to community users through the first 49 days after registration (all P values <.001). Two-step cluster analyses revealed 3 user groups in the RCT sample (Nonstarters, 10-Timers, and 30+-Timers) and 2 user groups in the community samples (2-Timers and 20-Timers). Groups seemed comparable in patterns of use but differed in magnitude, with RCT participant usage groups showing more frequent engagement than community usage groups. Only the high-usage group among RCT participants approached myCompass usage recommendations.
Findings suggested that external validity concerns of RCT designs may arise with regards to the predicted magnitude of eMH program use rather than overall usage styles. Following up RCT nonstarters may help provide unique insights into why individuals choose not to engage with an eMH program despite generally being willing to participate in an eMH evaluation study. Overestimating frequency of engagement with eMH tools may have theoretical implications and potentially impact economic considerations for plans to disseminate these tools to the general public.
采用强有力研究设计的随机对照试验（RCT）对于确定治疗效果至关重要。尽管 RCT 具有较高的内部有效性，但外部有效性问题限制了研究结果在一般人群中的推广。例如，当自我选择参加试验的研究参与者比其他个体更有动力遵守研究条件时，可能会引入偏差。这些外部有效性考虑因素扩展到电子心理健康（eMH）研究，特别是当 eMH 工具设计为公众访问并且提供最小或没有监督时。
采用聚类技术识别自我指导型 eMH 计划的 RCT 参与者和社区用户的参与情况。这种探索性方法检查了实际的而不是理论上的 RCT 参与者和社区用户的参与模式。这两个样本在同一时间段内都可以访问 eMH 计划，并在 eMH 计划网站上收到相同的使用建议。本研究的目的是帮助评估在评估和自然主义环境中，对电子心理健康工具使用行为的相似性和差异性的期望。
澳大利亚成年人注册了 myCompass，这是一种自我指导的在线治疗程序，旨在减轻轻度至中度的负面情绪症状。他们通过成为 RCT 入组的一部分（160/231，69.6%为女性）或在 2011 年 10 月至 2012 年 10 月期间在互联网上自由访问该程序（5563/8391，66.30%为女性）来注册。在注册时，RCT 参与者和社区用户提供了基本的人口统计学信息。系统记录了使用指标（登录次数、跟踪次数和学习活动次数）。
在注册时，样本在年龄上存在显著差异（P=.003），社区用户比 RCT 参与者平均年长 3 岁（平均值 41.78，标准差 13.64）。此外，与社区用户相比，RCT 参与者在所有使用指标上的计划使用频率在注册后 49 天内都更高（所有 P 值<.001）。两步聚类分析显示，RCT 样本中有 3 个用户组（非启动者、10 次使用者和 30+次使用者）和社区样本中有 2 个用户组（2 次使用者和 20 次使用者）。各组的使用模式似乎相似，但在规模上有所不同，RCT 参与者的使用组比社区使用组的参与度更高。只有 RCT 参与者中的高使用率组接近 myCompass 的使用建议。
研究结果表明，RCT 设计的外部有效性问题可能会出现在对 eMH 计划使用的预测幅度上，而不是整体使用方式上。对 RCT 的非启动者进行跟踪可能有助于提供有关为什么尽管一般愿意参与 eMH 评估研究，但个人选择不参与 eMH 计划的独特见解。高估电子心理健康工具的使用频率可能具有理论意义，并可能对向公众传播这些工具的计划的经济考虑产生影响。
