Dorosan Michael, Chen Ya-Lin, Zhuang Qingyuan, Lam Shao Wei Sean
Health Services Research Centre, Singapore Health Services Pte Ltd, Singapore, Singapore.
Department of Biomedical Informatics and Medical Education, University of Washington, Seattle, WA, United States.
JMIR Res Protoc. 2025 Jan 16;14:e63875. doi: 10.2196/63875.
Integrating algorithm-based clinical decision support (CDS) systems poses significant challenges in evaluating their actual clinical value. Such CDS systems are traditionally assessed via controlled but resource-intensive clinical trials.
This paper presents a review protocol for preimplementation in silico evaluation methods to enable broadened impact analysis under simulated environments before clinical trials.
We propose a scoping review protocol that follows an enhanced Arksey and O'Malley framework and PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines to investigate the scope and research gaps in the in silico evaluation of algorithm-based CDS models-specifically CDS decision-making end points and objectives, evaluation metrics used, and simulation paradigms used to assess potential impacts. The databases searched are PubMed, Embase, CINAHL, PsycINFO, Cochrane, IEEEXplore, Web of Science, and arXiv. A 2-stage screening process identified pertinent articles. The information extracted from articles was iteratively refined. The review will use thematic, trend, and descriptive analyses to meet scoping aims.
We conducted an automated search of the databases above in May 2023, with most title and abstract screenings completed by November 2023 and full-text screening extended from December 2023 to May 2024. Concurrent charting and full-text analysis were carried out, with the final analysis and manuscript preparation set for completion in July 2024. Publication of the review results is targeted from July 2024 to February 2025. As of April 2024, a total of 21 articles have been selected following a 2-stage screening process; these will proceed to data extraction and analysis.
We refined our data extraction strategy through a collaborative, multidisciplinary approach, planning to analyze results using thematic analyses to identify approaches to in silico evaluation. Anticipated findings aim to contribute to developing a unified in silico evaluation framework adaptable to various clinical workflows, detailing clinical decision-making characteristics, impact measures, and reusability of methods. The study's findings will be published and presented in forums combining artificial intelligence and machine learning, clinical decision-making, and health technology impact analysis. Ultimately, we aim to bridge the development-deployment gap through in silico evaluation-based potential impact assessments.
INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/63875.
整合基于算法的临床决策支持（CDS）系统在评估其实际临床价值方面面临重大挑战。传统上，此类CDS系统是通过可控但资源密集型的临床试验进行评估的。
本文提出了一项用于在计算机模拟环境中进行预实施评估方法的综述方案，以便在临床试验之前在模拟环境中进行更广泛的影响分析。
我们提出了一项范围综述方案，该方案遵循增强版的阿克斯和奥马利框架以及PRISMA-ScR（系统评价和元分析扩展的范围综述的首选报告项目）指南，以调查基于算法的CDS模型的计算机模拟评估中的范围和研究空白，特别是CDS决策终点和目标、使用的评估指标以及用于评估潜在影响的模拟范式。检索的数据库包括PubMed、Embase、CINAHL、PsycINFO、Cochrane、IEEEXplore、Web of Science和arXiv。采用两阶段筛选过程确定相关文章。从文章中提取的信息经过反复提炼。该综述将使用主题分析、趋势分析和描述性分析来实现范围综述目标。
我们于2023年5月对上述数据库进行了自动检索，大多数标题和摘要筛选于2023年11月完成，全文筛选从2023年12月延长至2024年5月。同时进行了图表绘制和全文分析，最终分析和稿件准备定于2024年7月完成。综述结果计划于2024年7月至2025年2月发表。截至2024年4月，经过两阶段筛选过程，共选出21篇文章；这些文章将进入数据提取和分析阶段。
我们通过协作、多学科的方法完善了数据提取策略，计划使用主题分析来分析结果，以确定计算机模拟评估的方法。预期的研究结果旨在有助于开发一个适用于各种临床工作流程的统一计算机模拟评估框架，详细说明临床决策特征、影响措施和方法的可重用性。该研究的结果将在结合人工智能和机器学习、临床决策以及卫生技术影响分析的论坛上发表和展示。最终，我们旨在通过基于计算机模拟评估的潜在影响评估来弥合开发与部署之间的差距。
国际注册报告识别号（IRRID）：DERR1-10.2196/63875。
