Behrens Michael R, Ruder Warren C
Department of Bioengineering, University of Pittsburgh; 300 Technology Drive, Pittsburgh, PA 15213, USA.
Department of Mechanical Engineering, Carnegie Mellon University; 5000 Forbes Ave. Pittsburgh, PA 15213, USA.
Adv Intell Syst. 2022 Oct;4(10). doi: 10.1002/aisy.202200023. Epub 2022 Jul 6.
Swimming microrobots are increasingly developed with complex materials and dynamic shapes and are expected to operate in complex environments in which the system dynamics are difficult to model and positional control of the microrobot is not straightforward to achieve. Deep reinforcement learning is a promising method of autonomously developing robust controllers for creating smart microrobots, which can adapt their behavior to operate in uncharacterized environments without the need to model the system dynamics. This article reports the development of a smart helical magnetic hydrogel microrobot that uses the soft actor critic reinforcement learning algorithm to autonomously derive a control policy which allows the microrobot to swim through an uncharacterized biomimetic fluidic environment under control of a time varying magnetic field generated from a three-axis array of electromagnets. The reinforcement learning agent learned successful control policies from both state vector input and raw images, and the control policies learned by the agent recapitulated the behavior of rationally designed controllers based on physical models of helical swimming microrobots. Deep reinforcement learning applied to microrobot control is likely to significantly expand the capabilities of the next generation of microrobots.
游泳微型机器人越来越多地采用复杂材料和动态形状进行开发，预计将在系统动力学难以建模且微型机器人的位置控制难以直接实现的复杂环境中运行。深度强化学习是一种很有前景的方法，可用于自主开发强大的控制器来创建智能微型机器人，这种机器人可以调整自身行为，在未表征的环境中运行，而无需对系统动力学进行建模。本文报道了一种智能螺旋磁性水凝胶微型机器人的开发，该机器人使用软 Actor-Critic 强化学习算法自主推导控制策略，使其能够在由三轴电磁铁阵列产生的时变磁场控制下，游过未表征的仿生流体环境。强化学习智能体从状态向量输入和原始图像中学习成功的控制策略，并且该智能体学习到的控制策略概括了基于螺旋游泳微型机器人物理模型的合理设计控制器的行为。应用于微型机器人控制的深度强化学习可能会显著扩展下一代微型机器人的能力。
