Department of Public Health and Primary Care, Leiden University Medical Centre, ZA Leiden, the Netherlands.
National eHealth Living Lab, Leiden University Medical Centre, ZA Leiden, the Netherlands.
JAMA Netw Open. 2024 Sep 3;7(9):e2432990. doi: 10.1001/jamanetworkopen.2024.32990.
The aging and multimorbid population and health personnel shortages pose a substantial burden on primary health care. While predictive machine learning (ML) algorithms have the potential to address these challenges, concerns include transparency and insufficient reporting of model validation and effectiveness of the implementation in the clinical workflow.
To systematically identify predictive ML algorithms implemented in primary care from peer-reviewed literature and US Food and Drug Administration (FDA) and Conformité Européene (CE) registration databases and to ascertain the public availability of evidence, including peer-reviewed literature, gray literature, and technical reports across the artificial intelligence (AI) life cycle.
PubMed, Embase, Web of Science, Cochrane Library, Emcare, Academic Search Premier, IEEE Xplore, ACM Digital Library, MathSciNet, AAAI.org (Association for the Advancement of Artificial Intelligence), arXiv, Epistemonikos, PsycINFO, and Google Scholar were searched for studies published between January 2000 and July 2023, with search terms that were related to AI, primary care, and implementation. The search extended to CE-marked or FDA-approved predictive ML algorithms obtained from relevant registration databases. Three reviewers gathered subsequent evidence involving strategies such as product searches, exploration of references, manufacturer website visits, and direct inquiries to authors and product owners. The extent to which the evidence for each predictive ML algorithm aligned with the Dutch AI predictive algorithm (AIPA) guideline requirements was assessed per AI life cycle phase, producing evidence availability scores.
The systematic search identified 43 predictive ML algorithms, of which 25 were commercially available and CE-marked or FDA-approved. The predictive ML algorithms spanned multiple clinical domains, but most (27 [63%]) focused on cardiovascular diseases and diabetes. Most (35 [81%]) were published within the past 5 years. The availability of evidence varied across different phases of the predictive ML algorithm life cycle, with evidence being reported the least for phase 1 (preparation) and phase 5 (impact assessment) (19% and 30%, respectively). Twelve (28%) predictive ML algorithms achieved approximately half of their maximum individual evidence availability score. Overall, predictive ML algorithms from peer-reviewed literature showed higher evidence availability compared with those from FDA-approved or CE-marked databases (45% vs 29%).
The findings indicate an urgent need to improve the availability of evidence regarding the predictive ML algorithms' quality criteria. Adopting the Dutch AIPA guideline could facilitate transparent and consistent reporting of the quality criteria that could foster trust among end users and facilitating large-scale implementation.
人口老龄化和多病共存以及卫生人员短缺给初级保健带来了巨大负担。虽然预测性机器学习 (ML) 算法有可能解决这些挑战，但人们关注的问题包括模型验证的透明度和报告不足，以及在临床工作流程中实施的有效性。
从同行评议文献以及美国食品和药物管理局 (FDA) 和欧洲合格评定 (CE) 注册数据库中系统地确定在初级保健中实施的预测性 ML 算法，并确定证据的公开可用性，包括人工智能 (AI) 生命周期内的同行评议文献、灰色文献和技术报告。
在 2000 年 1 月至 2023 年 7 月期间，在 PubMed、Embase、Web of Science、Cochrane 图书馆、Emcare、Academic Search Premier、IEEE Xplore、ACM Digital Library、MathSciNet、AAAI.org（人工智能协会）、arXiv、Epistemonikos、PsycINFO 和 Google Scholar 上搜索了研究，这些研究与 AI、初级保健和实施有关。搜索范围还包括从相关注册数据库中获得的 CE 标记或 FDA 批准的预测性 ML 算法。三名审查员收集了涉及产品搜索、参考文献探索、制造商网站访问以及直接向作者和产品所有者查询等策略的后续证据。根据 AI 生命周期阶段评估每个预测性 ML 算法的证据与荷兰 AI 预测算法 (AIPA) 指南要求的一致性程度，产生证据可用性分数。
系统搜索确定了 43 种预测性 ML 算法，其中 25 种是商业上可用的，并获得了 CE 标记或 FDA 批准。预测性 ML 算法涵盖多个临床领域，但大多数（27 [63%]）专注于心血管疾病和糖尿病。大多数（35 [81%]）是在过去 5 年内发表的。预测性 ML 算法的证据可用性因算法生命周期的不同阶段而异，第 1 阶段（准备）和第 5 阶段（影响评估）的证据报告最少（分别为 19%和 30%）。12 个（28%）预测性 ML 算法达到了其最大个人证据可用性评分的一半左右。总体而言，同行评议文献中的预测性 ML 算法比 FDA 批准或 CE 标记数据库中的算法具有更高的证据可用性（45%对 29%）。
研究结果表明，迫切需要提高预测性 ML 算法质量标准相关证据的可用性。采用荷兰 AIPA 指南可以促进质量标准的透明和一致报告，从而在最终用户中建立信任，并促进大规模实施。
