School of Mechanical Engineering, Shiraz University, Shiraz, Iran.
Department of IT and Computer Engineering, School of Electrical and Computer Engineering, Shiraz University, Shiraz, Iran.
Comput Biol Med. 2022 Sep;148:105860. doi: 10.1016/j.compbiomed.2022.105860. Epub 2022 Jul 14.
Type-2 diabetes mellitus is characterized by insulin resistance and impaired insulin secretion in the human body. Many endeavors have been made in terms of controlling and reducing blood glucose via the medium of automated controlling tools to increase precision and efficiency and reduce human error. Recently, reinforcement learning algorithms are proved to be powerful in the field of intelligent control, which was the motivation for the current study.
For the first time, a reinforcement algorithm called normalized advantage function (NAF) algorithm has been applied as a model-free reinforcement learning method to regulate the blood glucose level of type-2 diabetic patients through subcutaneous injection. The algorithm has been designed and developed in a model-free approach to avoid additional inaccuracies and parameter uncertainty introduced by the mathematical models of the glucoregulatory system. Insulin doses constitute the control action that is designed to be stated directly in clinical language with the unit IU. In this regard, a new environment state is considered in addition to the glucose level to take into account the delayed effect of insulin elimination under the skin. Finally, a simple but practical reward function is developed to be used with the NAF algorithm to correct the glucose level and maintain it in the desired range.
The simulation environment was set up to imitate the basal-bolus process accurately. Results for 30 days of simulation of the designed controller on three different average virtual patients verify the feasibility and effectiveness of the method and reveal our proposed controller's learning ability. Moreover, as the insulin elimination dynamic was taken into account, a more complete and more realistic model than the previously studied models has emerged.
NAF has proved a promising control approach, able to successfully regulate and significantly reduce the fluctuation of the blood glucose without meal announcements, compared to standard optimized open-loop basal-bolus therapies. The method and its results, which are directly in the clinical language, are applicable in real-time clinical situations.
2 型糖尿病的特征是人体的胰岛素抵抗和胰岛素分泌受损。人们通过自动化控制工具来控制和降低血糖，以提高精度和效率并减少人为错误，在这方面已经做出了许多努力。最近，强化学习算法在智能控制领域被证明具有强大的能力，这也是本研究的动机。
首次将一种称为归一化优势函数（NAF）算法的强化算法应用于通过皮下注射调节 2 型糖尿病患者的血糖水平的模型自由强化学习方法。该算法是在模型自由的方法中设计和开发的，以避免葡萄糖调节系统的数学模型引入的额外不准确性和参数不确定性。胰岛素剂量构成了控制作用，设计为直接用单位 IU 以临床语言表示。在这方面，除了葡萄糖水平外，还考虑了皮肤下胰岛素消除的延迟效应，引入了一个新的环境状态。最后，开发了一个简单但实用的奖励函数，与 NAF 算法一起用于校正血糖水平并将其维持在所需范围内。
模拟环境被设置为准确模拟基础-波状过程。对三个不同平均虚拟患者的设计控制器 30 天模拟的结果验证了该方法的可行性和有效性，并揭示了我们提出的控制器的学习能力。此外，由于考虑了胰岛素消除动力学，出现了比以前研究的模型更完整和更现实的模型。
与标准优化开环基础-波状疗法相比，NAF 已被证明是一种很有前途的控制方法，能够成功调节和显著降低血糖波动，而无需进食通知。该方法及其直接在临床语言中的结果适用于实时临床情况。
