Université de technologie de Compiègne, CNRS, Biomechanics and Bioengineering, Centre de Recherche Royallieu, CS 60 319-60 203, Compiègne, France.
Univ. Lille, CNRS, Centrale Lille, UMR 9013, LaMcube, Laboratoire de Mécanique, Multiphysique, Multiéchelle, 59655, Villeneuve d'Ascq Cedex, F-59000, Lille, France.
Med Biol Eng Comput. 2022 Jun;60(6):1745-1761. doi: 10.1007/s11517-022-02567-3. Epub 2022 Apr 22.
Reinforcement learning (RL) has been used to study human locomotion learning. One of the current challenges in healthcare is our understanding of and ability to slow the decline due to muscle ageing and its effect on human falls. The purpose of this study was to investigate reinforcement learning for human movement strategies when modifying muscle parameters to account for age-related changes. In particular, human falls with modified physiological factors were modelled and simulated to determine the effect of muscle descriptors for ageing on kinematic behaviour and muscle force control. A 3D musculoskeletal model (8 DoF and 22 muscles) of the human body was used. The deep deterministic policy gradient (DDPG) method was implemented. Different muscle descriptors for ageing were integrated, including changes in maximum isometric force, contraction velocity, the deactivation time constant and passive muscle strain. Additionally, the effects of isometric force reductions of 10, 20 and 30% were also considered independently. An environment for the simulation was developed using the opensim-rl package for Python with the training process completed on Google Compute Engine. The simulation outcomes for healthy young adult and elderly falls under modified muscle behaviours were compared to experimental observations for validation. The result of our elderly simulation for multiple ageing-related factors (M_all) produced a walking speed of 0.26 m/s for the two steps taken prior to the fall. The over activation of the hip extensors and inactivation of knee extensors led to a backward fall for this elderly simulation. The inactivated rectus femoris and right tibialis are main actors of the forward fall. Our simulation outcomes are consistent with experimental observations through the comparison of kinematic features and motion history evolution. We showed in the present study, for the first time, that RL can be used as a strategy to explore the effect of ageing muscle physiological factors on kinematics and muscle control during falls. Our findings show that the elderly fall model for the M_all condition more closely resembles experimental elderly fall data than our simulations which considered age-related reductions of force alone. As future perspectives, the behaviour preceding a fall will be studied to establish the strategies used to avoid falls or fall with minimal consequence, leading to the identification of patient-specific rehabilitation programmes for elderly people.
强化学习（RL）已被用于研究人类运动学习。当前医疗保健领域的一个挑战是，我们对肌肉老化及其对人类跌倒的影响的理解和减缓其影响的能力。本研究的目的是研究在修改肌肉参数以适应与年龄相关的变化时，人类运动策略的强化学习。特别是，对具有修改后的生理因素的人类跌倒进行了建模和模拟，以确定肌肉描述符对衰老对运动行为和肌肉力控制的影响。使用了人体的 3D 肌肉骨骼模型（8 自由度和 22 块肌肉）。实现了深度确定性策略梯度（DDPG）方法。整合了不同的衰老肌肉描述符，包括最大等长力、收缩速度、失活时间常数和被动肌肉应变的变化。此外，还分别考虑了等长力降低 10%、20%和 30%的影响。使用 Python 的 opensim-rl 包为模拟开发了一个环境，并在 Google Compute Engine 上完成了培训过程。将健康的年轻成年人和老年人跌倒在修改后的肌肉行为下的模拟结果与实验观察结果进行比较，以验证。对于多个与衰老相关的因素（M_all）的老年人模拟，模拟结果产生了跌倒前两步的 0.26 m/s 的步行速度。髋关节伸肌过度激活和膝关节伸肌失活导致了老年人模拟的向后跌倒。失活的股直肌和右胫骨前肌是向前跌倒的主要作用者。通过比较运动学特征和运动史演化，我们的模拟结果与实验观察结果一致。在本研究中，我们首次表明，RL 可以用作策略来探索衰老肌肉生理因素对跌倒时运动学和肌肉控制的影响。我们的研究结果表明，对于 M_all 条件的老年人跌倒模型，与仅考虑力的与年龄相关降低的模拟相比，更接近实验性老年人跌倒数据。作为未来的展望，将研究跌倒前的行为，以建立避免跌倒或跌倒时造成最小后果的策略，从而为老年人确定特定于患者的康复方案。
