Tejedor Miguel, Woldaregay Ashenafi Zebene, Godtliebsen Fred
Department of Computer Science, University of Tromsø-The Arctic University of Norway, Norway.
Department of Computer Science, University of Tromsø-The Arctic University of Norway, Norway.
Artif Intell Med. 2020 Apr;104:101836. doi: 10.1016/j.artmed.2020.101836. Epub 2020 Feb 21.
Reinforcement learning (RL) is a computational approach to understanding and automating goal-directed learning and decision-making. It is designed for problems which include a learning agent interacting with its environment to achieve a goal. For example, blood glucose (BG) control in diabetes mellitus (DM), where the learning agent and its environment are the controller and the body of the patient respectively. RL algorithms could be used to design a fully closed-loop controller, providing a truly personalized insulin dosage regimen based exclusively on the patient's own data.
In this review we aim to evaluate state-of-the-art RL approaches to designing BG control algorithms in DM patients, reporting successfully implemented RL algorithms in closed-loop, insulin infusion, decision support and personalized feedback in the context of DM.
An exhaustive literature search was performed using different online databases, analyzing the literature from 1990 to 2019. In a first stage, a set of selection criteria were established in order to select the most relevant papers according to the title, keywords and abstract. Research questions were established and answered in a second stage, using the information extracted from the articles selected during the preliminary selection.
The initial search using title, keywords, and abstracts resulted in a total of 404 articles. After removal of duplicates from the record, 347 articles remained. An independent analysis and screening of the records against our inclusion and exclusion criteria defined in Methods section resulted in removal of 296 articles, leaving 51 relevant articles. A full-text assessment was conducted on the remaining relevant articles, which resulted in 29 relevant articles that were critically analyzed. The inter-rater agreement was measured using Cohen Kappa test, and disagreements were resolved through discussion.
The advances in health technologies and mobile devices have facilitated the implementation of RL algorithms for optimal glycemic regulation in diabetes. However, there exists few articles in the literature focused on the application of these algorithms to the BG regulation problem. Moreover, such algorithms are designed for control tasks as BG adjustment and their use have increased recently in the diabetes research area, therefore we foresee RL algorithms will be used more frequently for BG control in the coming years. Furthermore, in the literature there is a lack of focus on aspects that influence BG level such as meal intakes and physical activity (PA), which should be included in the control problem. Finally, there exists a need to perform clinical validation of the algorithms.
强化学习（RL）是一种用于理解和自动化目标导向学习与决策的计算方法。它适用于包括学习智能体与环境交互以实现目标的问题。例如，糖尿病（DM）中的血糖（BG）控制，其中学习智能体和环境分别是控制器和患者的身体。RL算法可用于设计完全闭环控制器，仅根据患者自身数据提供真正个性化的胰岛素剂量方案。
在本综述中，我们旨在评估用于设计糖尿病患者BG控制算法的最新RL方法，报告在糖尿病背景下成功应用于闭环、胰岛素输注、决策支持和个性化反馈的RL算法。
使用不同的在线数据库进行了详尽的文献检索，分析了1990年至2019年的文献。在第一阶段，建立了一组选择标准，以便根据标题、关键词和摘要选择最相关的论文。在第二阶段，利用从初步筛选中选定文章提取的信息，确定并回答了研究问题。
最初使用标题、关键词和摘要进行检索共得到404篇文章。去除记录中的重复项后，还剩347篇文章。根据我们在方法部分定义的纳入和排除标准对记录进行独立分析和筛选，排除了296篇文章，剩下51篇相关文章。对其余相关文章进行了全文评估，最终对29篇相关文章进行了批判性分析。使用Cohen Kappa检验测量评分者间一致性，分歧通过讨论解决。
健康技术和移动设备的进步促进了RL算法在糖尿病最佳血糖调节中的应用。然而，文献中很少有文章关注这些算法在BG调节问题上的应用。此外，此类算法是为BG调整等控制任务设计的，最近在糖尿病研究领域的应用有所增加，因此我们预计未来几年RL算法将更频繁地用于BG控制。此外，文献中缺乏对影响BG水平的因素（如饮食摄入和身体活动（PA））的关注，而这些因素应纳入控制问题。最后，需要对算法进行临床验证。
