Ahmed Tamim, Thopalli Kowshik, Rikakis Thanassis, Turaga Pavan, Kelliher Aisling, Huang Jia-Bin, Wolf Steven L
Department of Biomedical Engineering, Virginia Tech, Blacksburg, VA, United States.
Geometric Media Lab, School of Arts, Media and Engineering, Arizona State University, Tempe, AZ, United States.
Front Neurol. 2021 Aug 19;12:720650. doi: 10.3389/fneur.2021.720650. eCollection 2021.
We are developing a system for long term Semi-Automated Rehabilitation At the Home (SARAH) that relies on low-cost and unobtrusive video-based sensing. We present a cyber-human methodology used by the SARAH system for automated assessment of upper extremity stroke rehabilitation at the home. We propose a hierarchical model for automatically segmenting stroke survivor's movements and generating training task performance assessment scores during rehabilitation. The hierarchical model fuses expert therapist knowledge-based approaches with data-driven techniques. The expert knowledge is more observable in the higher layers of the hierarchy (task and segment) and therefore more accessible to algorithms incorporating high level constraints relating to activity structure (i.e., type and order of segments per task). We utilize an HMM and a Decision Tree model to connect these high level priors to data driven analysis. The lower layers (RGB images and raw kinematics) need to be addressed primarily through data driven techniques. We use a transformer based architecture operating on low-level action features (tracking of individual body joints and objects) and a Multi-Stage Temporal Convolutional Network(MS-TCN) operating on raw RGB images. We develop a sequence combining these complimentary algorithms effectively, thus encoding the information from different layers of the movement hierarchy. Through this combination, we produce a robust segmentation and task assessment results on noisy, variable and limited data, which is characteristic of low cost video capture of rehabilitation at the home. Our proposed approach achieves 85% accuracy in per-frame labeling, 99% accuracy in segment classification and 93% accuracy in task completion assessment. Although the methodology proposed in this paper applies to upper extremity rehabilitation using the SARAH system, it can potentially be used, with minor alterations, to assist automation in many other movement rehabilitation contexts (i.e., lower extremity training for neurological accidents).
我们正在开发一种用于家庭长期半自动康复的系统（SARAH），该系统依赖于低成本且不引人注意的基于视频的传感技术。我们展示了SARAH系统用于家庭上肢中风康复自动评估的一种人机结合方法。我们提出了一种分层模型，用于在康复过程中自动分割中风幸存者的动作并生成训练任务表现评估分数。该分层模型将基于专家治疗师知识的方法与数据驱动技术相融合。专家知识在层次结构的较高层（任务和片段）中更易观察到，因此对于纳入与活动结构相关的高级约束（即每个任务的片段类型和顺序）的算法来说更容易获取。我们利用隐马尔可夫模型（HMM）和决策树模型将这些高级先验知识与数据驱动分析相连接。较低层（RGB图像和原始运动学数据）主要需要通过数据驱动技术来处理。我们使用一种基于变压器的架构来处理低级动作特征（跟踪个体身体关节和物体），以及一种多阶段时间卷积网络（MS-TCN）来处理原始RGB图像。我们有效地开发了一种结合这些互补算法的序列，从而对来自运动层次结构不同层的信息进行编码。通过这种结合，我们在嘈杂、可变且有限的数据上产生了强大的分割和任务评估结果，这是家庭康复低成本视频捕捉的特点。我们提出的方法在逐帧标注中达到了85%的准确率，在片段分类中达到了99%的准确率，在任务完成评估中达到了93%的准确率。尽管本文提出的方法适用于使用SARAH系统的上肢康复，但经过微小改动后，它有可能用于协助许多其他运动康复场景（即神经事故后的下肢训练）中的自动化。
