Institute of Integrated Traditional Chinese and Western Medicine, Chinese Evidence-Based Medicine and Cochrane China Center, West China Hospital, Sichuan University, Chengdu, 610041, China.
NMPA Key Laboratory for Real World Data Research and Evaluation in Hainan, Chengdu, 610041, China.
Mil Med Res. 2024 Aug 6;11(1):52. doi: 10.1186/s40779-024-00559-y.
In recent years, there has been a growing trend in the utilization of observational studies that make use of routinely collected healthcare data (RCD). These studies rely on algorithms to identify specific health conditions (e.g. diabetes or sepsis) for statistical analyses. However, there has been substantial variation in the algorithm development and validation, leading to frequently suboptimal performance and posing a significant threat to the validity of study findings. Unfortunately, these issues are often overlooked.
We systematically developed guidance for the development, validation, and evaluation of algorithms designed to identify health status (DEVELOP-RCD). Our initial efforts involved conducting both a narrative review and a systematic review of published studies on the concepts and methodological issues related to algorithm development, validation, and evaluation. Subsequently, we conducted an empirical study on an algorithm for identifying sepsis. Based on these findings, we formulated specific workflow and recommendations for algorithm development, validation, and evaluation within the guidance. Finally, the guidance underwent independent review by a panel of 20 external experts who then convened a consensus meeting to finalize it.
A standardized workflow for algorithm development, validation, and evaluation was established. Guided by specific health status considerations, the workflow comprises four integrated steps: assessing an existing algorithm's suitability for the target health status; developing a new algorithm using recommended methods; validating the algorithm using prescribed performance measures; and evaluating the impact of the algorithm on study results. Additionally, 13 good practice recommendations were formulated with detailed explanations. Furthermore, a practical study on sepsis identification was included to demonstrate the application of this guidance.
The establishment of guidance is intended to aid researchers and clinicians in the appropriate and accurate development and application of algorithms for identifying health status from RCD. This guidance has the potential to enhance the credibility of findings from observational studies involving RCD.
近年来，利用常规收集的医疗保健数据（RCD）进行观察性研究的趋势日益增长。这些研究依赖于算法来识别特定的健康状况（例如糖尿病或败血症）进行统计分析。然而，算法的开发和验证存在很大差异，导致性能经常不理想，并对研究结果的有效性构成重大威胁。不幸的是，这些问题往往被忽视。
我们系统地制定了用于开发、验证和评估旨在识别健康状况的算法的指南（DEVELOP-RCD）。我们的初步工作包括对与算法开发、验证和评估相关的概念和方法学问题进行叙述性综述和系统综述。随后，我们对用于识别败血症的算法进行了实证研究。基于这些发现，我们在指南中制定了算法开发、验证和评估的具体工作流程和建议。最后，该指南由 20 名外部专家组成的小组进行了独立审查，然后召开了共识会议对其进行了最终确定。
建立了算法开发、验证和评估的标准化工作流程。在特定健康状况考虑因素的指导下，工作流程包括四个集成步骤：评估现有算法对目标健康状况的适用性；使用推荐方法开发新算法；使用规定的性能指标验证算法；以及评估算法对研究结果的影响。此外，还制定了 13 条良好实践建议，并附有详细说明。此外，还包括一项关于败血症识别的实际研究，以展示该指南的应用。
该指南的建立旨在帮助研究人员和临床医生适当地、准确地开发和应用从 RCD 中识别健康状况的算法。该指南有可能提高涉及 RCD 的观察性研究结果的可信度。
