Yao Xiaomei, Saha Ashirbani, Saravanan Sharan, Low Ashley, Sussman Jonathan
Department of Health Research Methods, Evidence, and Impact McMaster University Hamilton Ontario Canada.
Department of Oncology McMaster University Hamilton Ontario Canada.
Cancer Innov. 2025 Jun 29;4(4):e70021. doi: 10.1002/cai2.70021. eCollection 2025 Aug.
Conducting a systematic review (SR) is a time-intensive process and represents the first phase in developing a clinical practice guideline (CPG). Completing a CPG through the Program in Evidence-Based Care (PEBC), a globally acknowledged guideline program supported by Ontario Health (Cancer Care Ontario), typically takes about 2 years. Thus, expediting an SR can significantly reduce the overall time required to complete a CPG. Our recently published review identified two artificial intelligence (AI) tools, DistillerSR and EPPI-Reviewer that reduced time in the title and abstract screening in an SR process when developing a CPG. However, the consistency and generalizability of these tools remain unclear within or across different SRs related to cancer. This study protocol aims to evaluate and compare the performance of DistillerSR and EPPI-Reviewer against human reviewers for title and abstract screening (Stage I screening) in cancer CPG development.
We will conduct a retrospective simulation study to evaluate and compare the performance of DistillerSR and EPPI-Reviewer across 10 previously published CPGs by PEBC. These CPGs include the five cancer types with the highest incidence (lung, breast, prostate, colorectal, and bladder). We will run 30 simulation trials for one CPG per AI tool. Primary outcomes are workload savings and time savings in Stage I screening. The secondary outcome is the percentage of missing articles among the final included articles. This informs the accuracy and comprehensiveness of the AI tools. Descriptive and inferential statistical analysis will be conducted to evaluate the outcomes.
This is a study protocol. The data presented in the tables are illustrative examples rather than actual study results, in accordance with the journal s standard structure. All data included in the final study will be thoroughly validated.
This will be the first study to investigate and compare the performance of DistillerSR and EPPI-Reviewer in Stage I screening of SRs in CPGs across different cancer types. These findings will inform the reliable use of AI tools in future cancer-related CPGs. The results from this retrospective study will need to be confirmed by prospective studies.
开展系统评价（SR）是一个耗时的过程，也是制定临床实践指南（CPG）的第一阶段。通过循证医疗项目（PEBC）完成一份CPG，该项目是一个得到安大略省卫生厅（安大略癌症护理机构）支持的全球公认的指南项目，通常需要大约两年时间。因此，加快系统评价能够显著减少完成一份CPG所需的总时间。我们最近发表的综述确定了两种人工智能（AI）工具，DistillerSR和EPPI-Reviewer，它们在制定CPG的系统评价过程中减少了标题和摘要筛选的时间。然而，在与癌症相关的不同系统评价中，这些工具的一致性和可推广性仍不明确。本研究方案旨在评估和比较DistillerSR和EPPI-Reviewer与人类评审员在癌症CPG制定中标题和摘要筛选（第一阶段筛选）的表现。
我们将进行一项回顾性模拟研究，以评估和比较DistillerSR和EPPI-Reviewer在PEBC之前发表的10份CPG中的表现。这些CPG包括发病率最高的五种癌症类型（肺癌、乳腺癌、前列腺癌、结直肠癌和膀胱癌）。每种人工智能工具针对一份CPG我们将进行30次模拟试验。主要结果是第一阶段筛选中的工作量节省和时间节省。次要结果是最终纳入文章中遗漏文章的百分比。这反映了人工智能工具的准确性和全面性。将进行描述性和推断性统计分析以评估结果。
这是一份研究方案。根据期刊的标准结构，表格中呈现的数据是示例而非实际研究结果。最终研究中包含的所有数据都将经过全面验证。
这将是第一项调查和比较DistillerSR和EPPI-Reviewer在不同癌症类型CPG的系统评价第一阶段筛选中表现的研究。这些发现将为未来与癌症相关的CPG中人工智能工具的可靠使用提供依据。这项回顾性研究的结果需要前瞻性研究来证实。
