Department of Biomedical Informatics, Emory University, Atlanta, GA, United States of America.
Department of Biomedical Engineering, Georgia Institute of Technology, Atlanta, GA, United States of America.
J Neural Eng. 2024 Jun 3;21(3):036027. doi: 10.1088/1741-2552/ad48bb.
. Vagus nerve stimulation (VNS) is being investigated as a potential therapy for cardiovascular diseases including heart failure, cardiac arrhythmia, and hypertension. The lack of a systematic approach for controlling and tuning the VNS parameters poses a significant challenge. Closed-loop VNS strategies combined with artificial intelligence (AI) approaches offer a framework for systematically learning and adapting the optimal stimulation parameters. In this study, we presented an interactive AI framework using reinforcement learning (RL) for automated data-driven design of closed-loop VNS control systems in a computational study.Multiple simulation environments with a standard application programming interface were developed to facilitate the design and evaluation of the automated data-driven closed-loop VNS control systems. These environments simulate the hemodynamic response to multi-location VNS using biophysics-based computational models of healthy and hypertensive rat cardiovascular systems in resting and exercise states. We designed and implemented the RL-based closed-loop VNS control frameworks in the context of controlling the heart rate and the mean arterial pressure for a set point tracking task. Our experimental design included two approaches; a general policy using deep RL algorithms and a sample-efficient adaptive policy using probabilistic inference for learning and control.Our simulation results demonstrated the capabilities of the closed-loop RL-based approaches to learn optimal VNS control policies and to adapt to variations in the target set points and the underlying dynamics of the cardiovascular system. Our findings highlighted the trade-off between sample-efficiency and generalizability, providing insights for proper algorithm selection. Finally, we demonstrated that transfer learning improves the sample efficiency of deep RL algorithms allowing the development of more efficient and personalized closed-loop VNS systems.We demonstrated the capability of RL-based closed-loop VNS systems. Our approach provided a systematic adaptable framework for learning control strategies without requiring prior knowledge about the underlying dynamics.
迷走神经刺激（VNS）作为一种治疗心力衰竭、心律失常和高血压等心血管疾病的潜在疗法正在研究中。缺乏控制和调整 VNS 参数的系统方法是一个重大挑战。闭环 VNS 策略与人工智能（AI）方法相结合，为系统地学习和适应最佳刺激参数提供了框架。在这项研究中，我们提出了一种使用强化学习（RL）的交互式 AI 框架，用于在计算研究中自动数据驱动设计闭环 VNS 控制系统。开发了多个具有标准应用程序编程接口的模拟环境，以方便自动化数据驱动闭环 VNS 控制系统的设计和评估。这些环境使用基于生物物理学的健康和高血压大鼠心血管系统计算模型模拟多部位 VNS 对血液动力学的影响，处于休息和运动状态。我们在设定点跟踪任务中设计并实现了基于 RL 的闭环 VNS 控制框架，以控制心率和平均动脉压。我们的实验设计包括两种方法；一种是使用深度 RL 算法的通用策略，另一种是使用概率推理进行学习和控制的样本高效自适应策略。我们的模拟结果表明，闭环基于 RL 的方法能够学习最佳的 VNS 控制策略，并适应目标设定点和心血管系统底层动力学的变化。我们的发现强调了样本效率和通用性之间的权衡，为正确的算法选择提供了见解。最后，我们证明了迁移学习可以提高深度 RL 算法的样本效率，从而开发更高效和个性化的闭环 VNS 系统。我们展示了基于 RL 的闭环 VNS 系统的能力。我们的方法提供了一种系统的、适应性强的框架，用于学习控制策略，而无需事先了解底层动态。
