Cai Mingxue, Wang Qianqian, Qi Zhaoyang, Jin Dongdong, Wu Xinyu, Xu Tiantian, Zhang Li
IEEE Trans Cybern. 2023 Dec;53(12):7699-7711. doi: 10.1109/TCYB.2022.3199213. Epub 2023 Nov 29.
Soft magnetic miniature robots (SMMRs) have potential biomedical applications due to their flexible size and mobility to access confined environments. However, navigating the robot to a goal site with precise control performance and high repeatability in unstructured environments, especially in flow rate conditions, still remains a challenge. In this study, drawing inspiration from the control requirements of drug delivery and release to the goal lesion site in the presence of dynamic biofluids, we propose a flow rate rejection control strategy based on a deep reinforcement learning (DRL) framework to actuate an SMMR to achieve goal-reaching and hovering in fluidic tubes. To this end, an SMMR is first fabricated, which can be operated by an external magnetic field to realize its desired functionalities. Subsequently, a simulator is constructed based on neural networks to map the relationship between the applied magnetic field and robot locomotion states. With minimal prior knowledge about the environment and dynamics, a gated recurrent unit (GRU)-based DRL algorithm is formulated by considering the designed history state-action and estimated flow rates. In addition, the randomization technique is applied during training to distill the general control policy for the physical SMMR. The results of numerical simulations and experiments are illustrated to demonstrate the robustness and efficacy of the presented control framework. Finally, in-depth analyses and discussions indicate the potentiality of DRL for soft magnetic robots in biomedical applications.
软磁微型机器人（SMMRs）因其灵活的尺寸和进入受限环境的移动性而具有潜在的生物医学应用前景。然而，在非结构化环境中，尤其是在流速条件下，将机器人精确控制并高重复性地导航到目标位置仍然是一个挑战。在本研究中，从在动态生物流体存在的情况下将药物输送和释放到目标病变部位的控制要求中获得灵感，我们提出了一种基于深度强化学习（DRL）框架的流速抑制控制策略，以驱动软磁微型机器人在流体管道中实现到达目标和悬停。为此，首先制造了一个软磁微型机器人，它可以通过外部磁场操作以实现其期望的功能。随后，基于神经网络构建了一个模拟器，以映射施加的磁场与机器人运动状态之间的关系。在对环境和动力学的先验知识最少的情况下，通过考虑设计的历史状态-动作和估计的流速，制定了一种基于门控循环单元（GRU）的深度强化学习算法。此外，在训练期间应用随机化技术来提炼物理软磁微型机器人的通用控制策略。数值模拟和实验结果表明了所提出控制框架的鲁棒性和有效性。最后，深入的分析和讨论表明了深度强化学习在软磁机器人生物医学应用中的潜力。
