Archila John, Manzanera Antoine, Martínez Fabio
Biomedical Imaging, Vision and Learning Laboratory(BivL2ab), Universidad Industrial de Santander (UIS), Bucaramanga, 680002 Santander Colombia.
Unité d'Informatique et d'Ingénierie des Systèmes (U2IS), ENSTA Paris, Institut Polytechnique de Paris, Palaiseau, 91120 Essonne France.
Biomed Eng Lett. 2024 Oct 26;15(1):81-93. doi: 10.1007/s13534-024-00420-0. eCollection 2025 Jan.
Parkinson's disease is a neurodegenerative disorder principally manifested as motor disabilities. In clinical practice, diagnostic rating scales are available for broadly measuring, classifying, and characterizing the disease progression. Nonetheless, these scales depend on the specialist's expertise, introducing a high degree of subjectivity. Thus, diagnosis and motor stage identification may be affected by misinterpretation, leading to incorrect or misguided treatments. This work addresses how to learn multimodal representations based on compact gait and eye motion descriptors whose fusion improves disease diagnosis prediction. This work introduces a noninvasive multimodal strategy that combines gait and ocular pursuit motion modalities into a geometrical Riemannian Neural Network for PD quantification and diagnostic support. Markerless gait and ocular pursuit videos were first recorded as Parkinson's observations, which are represented at each frame by a set of frame convolutional deep features. Then, Riemannian means are computed per modality using frame-level covariances coded from convolutional deep features. Thus, a geometrical learning representation is adjusted by Riemannian means, following early, intermediate, and late fusion alternatives. The adjusted Riemannian manifold combines input modalities to obtain PD prediction. The geometrical multimodal approach was validated in a study involving 13 control subjects and 19 PD patients, achieving a mean accuracy of 96% for early and intermediate fusion and 92% for late fusion, increasing the unimodal accuracy results obtained in the gait and eye movement modalities by 6 and 8%, respectively. The proposed method was able to discriminate Parkinson's patients from healthy subjects using multimodal geometrical configurations based on covariances descriptors. The covariance representation of video descriptors is highly compact (with an input size of 625 and an output size of 256 (1 BiRe)), facilitating efficient learning with a small number of samples, a crucial aspect in medical applications.
帕金森病是一种主要表现为运动功能障碍的神经退行性疾病。在临床实践中，有诊断评分量表可用于广泛测量、分类和描述疾病进展。然而，这些量表依赖于专家的专业知识，引入了高度的主观性。因此，诊断和运动阶段识别可能会受到错误解读的影响，导致不正确或误导性的治疗。这项工作探讨了如何基于紧凑的步态和眼动描述符学习多模态表示，其融合可改善疾病诊断预测。这项工作引入了一种非侵入性多模态策略，该策略将步态和眼球追踪运动模态结合到一个几何黎曼神经网络中，用于帕金森病的量化和诊断支持。首先记录无标记步态和眼球追踪视频作为帕金森病的观察数据，这些数据在每一帧由一组帧卷积深度特征表示。然后，使用从卷积深度特征编码的帧级协方差为每个模态计算黎曼均值。因此，通过黎曼均值调整几何学习表示，遵循早期、中期和晚期融合方案。调整后的黎曼流形结合输入模态以获得帕金森病预测。在一项涉及13名对照受试者和19名帕金森病患者的研究中验证了几何多模态方法，早期和中期融合的平均准确率达到96%，晚期融合的平均准确率达到92%，分别比步态和眼动模态中获得的单模态准确率结果提高了6%和8%。所提出的方法能够使用基于协方差描述符的多模态几何配置将帕金森病患者与健康受试者区分开来。视频描述符的协方差表示非常紧凑（输入大小为625，输出大小为256（1个BiRe）），便于用少量样本进行高效学习，这在医学应用中是一个关键方面。
