Department of Engineering and Public Policy, Carnegie Mellon University, Pittsburgh, PA, United States.
Division of General Internal Medicine, University of Pittsburgh, Pittsburgh, PA, United States.
JMIR Mhealth Uhealth. 2023 Nov 15;11:e46237. doi: 10.2196/46237.
The rapid growth of digital health apps has necessitated new regulatory approaches to ensure compliance with safety and effectiveness standards. Nonadherence and heterogeneous user engagement with digital health apps can lead to trial estimates that overestimate or underestimate an app's effectiveness. However, there are no current standards for how researchers should measure adherence or address the risk of bias imposed by nonadherence through efficacy analyses.
This systematic review aims to address 2 critical questions regarding clinical trials of software as a medical device (SaMD) apps: How well do researchers report adherence and engagement metrics for studies of effectiveness and efficacy? and What efficacy analyses do researchers use to account for nonadherence and how appropriate are their methods?
We searched the Food and Drug Administration's registration database for registrations of repeated-use, patient-facing SaMD therapeutics. For each such registration, we searched ClinicalTrials.gov, company websites, and MEDLINE for the corresponding clinical trial and study articles through March 2022. Adherence and engagement data were summarized for each of the 24 identified articles, corresponding to 10 SaMD therapeutics. Each article was analyzed with a framework developed using the Cochrane risk-of-bias questions to estimate the potential effects of imperfect adherence on SaMD effectiveness. This review, funded by the Richard King Mellon Foundation, is registered on the Open Science Framework.
We found that although most articles (23/24, 96%) reported collecting information about SaMD therapeutic engagement, of the 20 articles for apps with prescribed use, only 9 (45%) reported adherence information across all aspects of prescribed use: 15 (75%) reported metrics for the initiation of therapeutic use, 16 (80%) reported metrics reporting adherence between the initiation and discontinuation of the therapeutic (implementation), and 4 (20%) reported the discontinuation of the therapeutic (persistence). The articles varied in the reported metrics. For trials that reported adherence or engagement, there were 4 definitions of initiation, 8 definitions of implementation, and 4 definitions of persistence. All articles studying a therapeutic with a prescribed use reported effectiveness estimates that might have been affected by nonadherence; only a few (2/20, 10%) used methods appropriate to evaluate efficacy.
This review identifies 5 areas for improving future SaMD trials and studies: use consistent metrics for reporting adherence, use reliable adherence metrics, preregister analyses for observational studies, use less biased efficacy analysis methods, and fully report statistical methods and assumptions.
数字健康应用程序的快速增长需要新的监管方法来确保符合安全性和有效性标准。数字健康应用程序的不依从和异质用户参与可能导致试验估计值过高或过低估计应用程序的有效性。然而，目前还没有研究人员应该如何衡量依从性或通过疗效分析解决不依从性带来的偏倚风险的标准。
本系统评价旨在解决软件即医疗设备 (SaMD) 应用程序临床试验的 2 个关键问题：研究人员对有效性和疗效研究报告依从性和参与度指标的情况如何？以及研究人员使用何种疗效分析来考虑不依从性，其方法是否合适？
我们搜索了食品和药物管理局的重复使用、面向患者的 SaMD 治疗剂注册数据库。对于每个此类注册，我们通过 2022 年 3 月在 ClinicalTrials.gov、公司网站和 MEDLINE 中搜索了相应的临床试验和研究文章。总结了 24 篇已识别文章中的 24 篇文章的依从性和参与度数据，对应 10 种 SaMD 治疗剂。使用基于 Cochrane 偏倚问题的框架分析了每篇文章，以估计不依从对 SaMD 有效性的潜在影响。这项由理查德·金·梅隆基金会资助的研究在开放科学框架上注册。
我们发现，尽管大多数文章（24 篇中的 23 篇，96%）报告了收集 SaMD 治疗参与信息，但在 20 篇规定用途应用程序的文章中，只有 9 篇（45%）报告了规定用途所有方面的依从性信息：15 篇（75%）报告了治疗开始的度量标准，16 篇（80%）报告了治疗开始和治疗结束之间的依从性报告（实施），4 篇（20%）报告了治疗的结束（坚持）。文章在报告的指标上存在差异。对于报告依从性或参与度的试验，有 4 种启动定义、8 种实施定义和 4 种坚持定义。所有研究规定用途治疗的文章都报告了可能受到不依从性影响的有效性估计值；只有少数（20 篇中的 2 篇，10%）使用了评估疗效的适当方法。
本评价确定了未来 SaMD 试验和研究需要改进的 5 个方面：使用一致的报告依从性指标、使用可靠的依从性指标、为观察性研究预先注册分析、使用偏差较小的疗效分析方法、以及充分报告统计方法和假设。
