Trella Anna L, Zhang Kelly W, Nahum-Shani Inbal, Shetty Vivek, Doshi-Velez Finale, Murphy Susan A
Department of Computer Science, Harvard University.
Institute for Social Research, University of Michigan.
Proc Innov Appl Artif Intell Conf. 2023 Jun 27;37(13):15724-15730. doi: 10.1609/aaai.v37i13.26866.
While dental disease is largely preventable, professional advice on optimal oral hygiene practices is often forgotten or abandoned by patients. Therefore patients may benefit from timely and personalized encouragement to engage in oral self-care behaviors. In this paper, we develop an online reinforcement learning (RL) algorithm for use in optimizing the delivery of mobile-based prompts to encourage oral hygiene behaviors. One of the main challenges in developing such an algorithm is ensuring that the algorithm considers the impact of current actions on the effectiveness of future actions (i.e., delayed effects), especially when the algorithm has been designed to run stably and autonomously in a constrained, real-world setting characterized by highly noisy, sparse data. We address this challenge by designing a quality reward that maximizes the desired health outcome (i.e., high-quality brushing) while minimizing user burden. We also highlight a procedure for optimizing the hyperparameters of the reward by building a simulation environment test bed and evaluating candidates using the test bed. The RL algorithm discussed in this paper will be deployed in Oralytics. To the best of our knowledge, Oralytics is the first mobile health study utilizing an RL algorithm designed to prevent dental disease by optimizing the delivery of motivational messages supporting oral self-care behaviors.
虽然牙科疾病在很大程度上是可以预防的，但患者往往会忘记或摒弃关于最佳口腔卫生习惯的专业建议。因此，患者可能会从及时且个性化的鼓励中受益，从而参与口腔自我护理行为。在本文中，我们开发了一种在线强化学习（RL）算法，用于优化基于移动设备的提示信息传递，以鼓励口腔卫生行为。开发这种算法的主要挑战之一是确保算法考虑当前行动对未来行动有效性的影响（即延迟效应），尤其是当算法被设计为在以高度嘈杂、稀疏数据为特征的受限现实环境中稳定且自主运行时。我们通过设计一种质量奖励来应对这一挑战，该奖励在最小化用户负担的同时，使期望的健康结果（即高质量刷牙）最大化。我们还强调了一种通过构建模拟环境测试平台并使用该平台评估候选参数来优化奖励超参数的程序。本文讨论的RL算法将部署在Oralytics中。据我们所知，Oralytics是第一项利用RL算法的移动健康研究，该算法旨在通过优化支持口腔自我护理行为的激励信息传递来预防牙科疾病。
