Hart Steven N, Day Patrick L, Garcia Christopher A
Department of Laboratory Medicine and Pathology, Mayo Clinic, 200 1st St. SW, Rochester, MN, 55901, USA.
BMC Med Inform Decis Mak. 2025 Jul 8;25(1):254. doi: 10.1186/s12911-025-03099-0.
Developing medical software requires navigating complex regulatory, ethical, and operational challenges. A comprehensive framework that supports both technical maturity and clinical safety is essential for effective artificial intelligence and machine learning system deployment. This paper introduces the Clinical Artificial Intelligence Readiness Evaluator Lifecycle and the Clinical Artificial Intelligence Readiness Evaluator Agent-a framework and AI-driven tool designed to streamline technology readiness level assessments in medical software development.
We developed the framework using an iterative process grounded in collaborative stakeholder analysis. Key institutional stakeholders-including clinical informatics experts, data engineers, ethicists, and operational leaders-were engaged to identify and prioritize the regulatory, ethical, and technical requirements unique to clinical AI/ML development. This approach, combined with a thorough review of existing methodologies, informed the creation of a lifecycle model that guides technology maturation from initial concept to full deployment. The AI-driven tool was implemented using a retrieval-augmented generation strategy and evaluated through a synthetic use case (the Diabetes Outcome Predictor). Evaluation metrics included the proportion of correctly addressed assessment questions and the overall time required for automated review, with human adjudication validating the tool's performance.
The findings indicate that the proposed framework effectively captures the complexities of clinical AI development. In the synthetic use case, the AI-driven tool identified that 32.8% of the assessment questions remained unanswered, while human adjudication confirmed discrepancies in 19.4% of these instances. These outcomes suggest that, when fully refined, the automated assessment process can reduce the need for extensive multi-stakeholder involvement, accelerate project timelines, and enhance resource efficiency.
The Clinical Artificial Intelligence Readiness Evaluator Lifecycle and Agent offer a robust and methodologically sound approach for evaluating the maturity of medical AI systems. By integrating stakeholder-driven insights with an AI-based assessment process, this framework lays the groundwork for more streamlined, secure, and effective clinical AI development. Future work will focus on optimizing retrieval strategies and expanding validation across diverse clinical applications.
开发医疗软件需要应对复杂的监管、伦理和运营挑战。一个支持技术成熟度和临床安全性的综合框架对于有效的人工智能和机器学习系统部署至关重要。本文介绍了临床人工智能就绪评估器生命周期和临床人工智能就绪评估器代理——一个旨在简化医疗软件开发中技术就绪水平评估的框架和人工智能驱动工具。
我们采用基于利益相关者协作分析的迭代过程开发了该框架。关键的机构利益相关者，包括临床信息学专家、数据工程师、伦理学家和运营领导者，参与确定临床人工智能/机器学习开发特有的监管、伦理和技术要求，并对其进行优先级排序。这种方法，结合对现有方法的全面审查，为创建一个生命周期模型提供了信息，该模型指导技术从最初概念到全面部署的成熟过程。人工智能驱动工具采用检索增强生成策略实施，并通过一个综合用例（糖尿病结果预测器）进行评估。评估指标包括正确回答的评估问题的比例以及自动审查所需的总时间，由人工裁决验证工具的性能。
研究结果表明，所提出的框架有效地捕捉了临床人工智能开发的复杂性。在综合用例中，人工智能驱动工具识别出32.8%的评估问题未得到回答，而人工裁决在这些情况中的19.4%中确认存在差异。这些结果表明，经过充分完善后，自动评估过程可以减少广泛的多利益相关者参与的需求，加快项目进度，并提高资源效率。
临床人工智能就绪评估器生命周期和代理为评估医疗人工智能系统的成熟度提供了一种强大且方法合理的方法。通过将利益相关者驱动的见解与基于人工智能的评估过程相结合，该框架为更简化、安全和有效的临床人工智能开发奠定了基础。未来的工作将集中在优化检索策略以及在各种临床应用中扩大验证范围。
