UKRI Centre for Doctoral Training in AI for Healthcare, Imperial College London, London, UK.
Brain & Behvaiour Lab: Departments of Bioengineering and Computing, Imperial College London, London, UK.
BMJ Health Care Inform. 2022 Jul;29(1). doi: 10.1136/bmjhci-2022-100549.
Establishing confidence in the safety of Artificial Intelligence (AI)-based clinical decision support systems is important prior to clinical deployment and regulatory approval for systems with increasing autonomy. Here, we undertook safety assurance of the AI Clinician, a previously published reinforcement learning-based treatment recommendation system for sepsis.
As part of the safety assurance, we defined four clinical hazards in sepsis resuscitation based on clinical expert opinion and the existing literature. We then identified a set of unsafe scenarios, intended to limit the action space of the AI agent with the goal of reducing the likelihood of hazardous decisions.
Using a subset of the Medical Information Mart for Intensive Care (MIMIC-III) database, we demonstrated that our previously published 'AI clinician' recommended fewer hazardous decisions than human clinicians in three out of our four predefined clinical scenarios, while the difference was not statistically significant in the fourth scenario. Then, we modified the reward function to satisfy our safety constraints and trained a new AI Clinician agent. The retrained model shows enhanced safety, without negatively impacting model performance.
While some contextual patient information absent from the data may have pushed human clinicians to take hazardous actions, the data were curated to limit the impact of this confounder.
These advances provide a use case for the systematic safety assurance of AI-based clinical systems towards the generation of explicit safety evidence, which could be replicated for other AI applications or other clinical contexts, and inform medical device regulatory bodies.
在具有越来越高自主性的系统获得临床部署和监管批准之前，建立对基于人工智能（AI）的临床决策支持系统的安全性信心非常重要。在这里，我们对之前发表的基于强化学习的脓毒症治疗推荐系统 AI 临床医生进行了安全性保证。
作为安全性保证的一部分，我们根据临床专家意见和现有文献定义了脓毒症复苏中的四个临床危害。然后，我们确定了一组不安全场景，旨在限制 AI 代理的动作空间，以降低危险决策的可能性。
使用医疗信息市场（MIMIC-III）数据库的一个子集，我们证明了我们之前发表的“AI 临床医生”在我们定义的四个临床场景中的三个场景中比人类临床医生推荐的危险决策更少，而在第四个场景中差异不具有统计学意义。然后，我们修改了奖励函数以满足我们的安全约束，并训练了一个新的 AI 临床医生代理。经过重新训练的模型显示出增强的安全性，而不会对模型性能产生负面影响。
虽然数据中缺少一些上下文患者信息可能促使人类临床医生采取危险行动，但数据经过精心筛选以限制这种混杂因素的影响。
这些进展为基于 AI 的临床系统的系统安全性保证提供了一个用例，以生成明确的安全性证据，这可以复制到其他 AI 应用或其他临床环境中，并为医疗设备监管机构提供信息。
