School of Computer Science and Technology, Dalian University of Technology, No. 2, Linggong Road, Dalian, 116024, China.
Department of Computer Science, Hong Kong Baptist University, Kowloon Tong, Hong Kong, China.
BMC Med Inform Decis Mak. 2019 Apr 9;19(Suppl 2):60. doi: 10.1186/s12911-019-0755-6.
Reinforcement learning (RL) provides a promising technique to solve complex sequential decision making problems in health care domains. However, existing studies simply apply naive RL algorithms in discovering optimal treatment strategies for a targeted problem. This kind of direct applications ignores the abundant causal relationships between treatment options and the associated outcomes that are inherent in medical domains.
This paper investigates how to integrate causal factors into an RL process in order to facilitate the final learning performance and increase explanations of learned strategies. A causal policy gradient algorithm is proposed and evaluated in dynamic treatment regimes (DTRs) for HIV based on a simulated computational model.
Simulations prove the effectiveness of the proposed algorithm for designing more efficient treatment protocols in HIV, and different definitions of the causal factors could have significant influence on the final learning performance, indicating the necessity of human prior knowledge on defining a suitable causal relationships for a given problem.
More efficient and robust DTRs for HIV can be derived through incorporation of causal factors between options of anti-HIV drugs and the associated treatment outcomes.
强化学习（RL）为解决医疗领域复杂的序贯决策问题提供了一种很有前途的技术。然而，现有的研究只是简单地将原始的 RL 算法应用于发现针对特定问题的最佳治疗策略。这种直接应用忽略了医疗领域中治疗方案与相关结果之间固有的丰富因果关系。
本文研究了如何将因果因素纳入 RL 过程中，以提高最终的学习性能和增强对所学策略的解释能力。基于模拟计算模型，针对 HIV 提出并评估了一种因果策略梯度算法。
模拟证明了所提出的算法在设计更有效的 HIV 治疗方案方面的有效性，并且因果因素的不同定义对最终的学习性能有显著影响，这表明对于给定问题，人为定义合适的因果关系的先验知识是必要的。
通过在抗 HIV 药物的选择与相关治疗结果之间引入因果因素，可以得出更有效和稳健的 HIV 动态治疗方案。
